Derik in Minnesota

Like bees gathering honey.

2009-07-09T22:53:19Z

Sometimes the human race is pretty damn amazing.

This is the near-Earth view– mostly weather and scientific satellites. The communications satellites orbit much further out.
Sped up ~ 2000x.

It’s pretty. And they almost never collide, using ballistic orbits.

We can do almost anything… it’s making us want to that’s the trick.

The Bastards of Free Culture

2009-07-08T05:42:01Z

I’m a Transformers fan. For the last few years I’ve been active on the Transformers Wiki. And the TFWiki is currently debating whether or not to migrate our site-wide licensing from GFDL to CC-BY-SA3. Essentially a platform change in our site’s underlying “legalese” from Linux to Mac.

Sounds boring, right?

The complete meltdown of Western Civilization, after the cut.

The New GNU

On June 15, Wikipedia changed from a GFDL license to Creative Commons license, that’s the legalese governing what can and cannot be done with their content. The Creative Commons license is a sort of “human readable” legal boilerplate created in 2002.

Jimbo Wales (founder of Wikipedia) is on-record as saying that if Creative Commons had been available when Wikipedia launched in 2001, it would have been the license Wikipedia used. But since it didn’t exist yet, they were forced to use the GNU Free Documentation License, a license designed for “manuals, textbooks, other reference and instructional materials, and documentation which often accompanies GNU software.”

The Creative Commons license has many ‘flavors,’ one of which is “CC-BY-SA3,” or “Creative Commons — Attribution-Share Alike 3.0,” whose general terms are more-or-less equivalent to that of the GFDL;

Anyone can make a copy of or take and alter this work.
The resulting work must also be freely available for others to copy or alter under the same terms.
The original authors must be credited.

GFDL has a bunch of clauses and conditions relating to invariant sections and cover texts such as you might encounter with software manuals. Wikipedia simplifies this by not having any of those, allowing it to pretty much ignore half the license as not relevant to its content.

While GFDL was workable, it wasn’t designed with the type of content-generation / user-contribution-model used by Wikipedia in mind; so there was understandable pressure to switch to a new option.

What does “Reliscensing” mean?

To relicense does not mean “change the license,” as you might think… ‘re’ here means ‘again’; the old license is not discarded.
Content offered under GFDL has the legal requirement that all future versions of that content also be available under GFDL. No matter what is done to it, the content must remain available for anyone else to re-use under the terms of the GFDL.

Simply switching your wiki from GFDL to a Creative Commons license would not normally remove the GFDL requirement… it would mean that the content was available under both GFDL and CC, leaving it up to the person re-using the content to decide which license’s terms they prefer to be bound by.

This is very silly; GFDL and CC-BY-SA3 both have essentially the same rights and requirements. There’s little practical difference between them. Offering the content under both licenses is very much like McDonalds listing a “large soda” and a “32 ounce soda” and allowing customers to choose which they’d prefer. A large soda is 32 ounces. You’d get two of the same item, but with a receipt treating them like they were completely different things.

An amendment was added to GFDL 1.3 in 2007 that released wikis and similar “massively multi-user collaborations” from the requirement that all future content be GFDL, provided that they switch to the nigh-identical CC-BY-SA3 before August 1, 2009. GFDL is a forward-compatible license, so they can do that, just like a board or directors can conven a meeting to change the company charter. Such changes apply to all GFDL content, even stuff createdlicensed under older versions of the License. (You can opt-out of the forward-computability clause that allows the FSF to revise the license on you, but no one bothers to.)

Under this arrangement, Wikipedia’s article on “Gold Farming” as it existed on June 14 remains available for use under GFDL or Creative Commons. The version of the text prior to the changeover could be moved to any GFDL project (such as a private wiki) or a Creative Commons project with no legal problems. But the Gold Farming article has (as of this writing) been edited 6 times since the July 16th legal changeover. And the version of the article which includes those changes is covered by CC-BY-SA3 only. So it cannot be incorporated into a project whose content is GFDL– the switch was one-way and one-time, (before August 1.) Going forward, all future versions of the article will be Creative Commons only.

This is how GNU puts it. (Note: the following excerpt from GFDL 1.3 has been edited for clarity. Certian clauses and legal terms have been replaced in order to present the relevant text specifically as is applies to wikis. The struck-out section does not apply to wikis, but was left in because it looked funny without it.)

11. RELICENSING
<snip>
A wiki is “eligible for relicensing” if it is licensed under GFDL, and if any content originally published under GFDL somewhere other than this wiki and later wholly or partially incorporated into it; (1) ~~had no cover texts or invariant sections~~, and (2) were thus incorporated prior to November 1, 2008.

The operator of a wiki may republish a wiki’s contents under CC-BY-SA on the same site at any time before August 1, 2009, provided the wiki is eligible for relicensing as described above.

GFDL vs. CC-BY-SA3

Back when the Transformers Wiki was decamping from Wikia, I had the singular joy of reviewing the text of Wikia’s Terms of Service and the GFDL in order to figure out just what had to be done for the move. (And specifically to figure out if we could dump Wikia’s annoying link-export spam. Answer: Yes.)

Because anything worth doing is worth doing right (and slowly) I took the time to review the complete text of the GFDL (all 3 versions) and CC-BY-SA3, with a specific eye toward how they apply to wikis. This is what I’ve learned;

The common assertion that the Creative Commons License is easier to understand is a lie

Creative Commons has an easy-to-understand graphical front-end that tells you what you can do with it. But as they note at the end… This is a human-readable summary of the Legal Code.
This front-end carries no legal weight, it’s just a summary. And the legal status it describes doesn’t even match up with the legalese in places! (More on that later.)
GFDL has a non-binding “Section 0″ that it uses for a mission statement, but it’s clearly labeled as such. The Creative Commons GUI presents a real danger of confusing users into believing the simplified GUI itself is legally binding.
The actual legalese of CC-BY-SA3 is denser and harder to read than GFDL 1.3… but it’s also a bit shorter. On balance… GFDL is still the easier to understand document, even with all the stuff that’s irrelevant to Wikis… but not by much.
See Attached License

The GFDL requires that any copy or derivative of the work include the full-text of the GFDL license itself. Not a big deal with electronic copies– you just provide a link. But if you wanted to excerpt any significant chunk of an article’s text for a book, you’d also have to publish a 3-page legal license.

Creative Commons allows re-users to simply note that the work is licensed under the the Creative Commons CC-BY-SA3 license. A significant plus as Wikipedia becomes more and more cited and excerpted. CC also dumps on all the provisions for cover-sheets and invariant sections that don’t apply to wikis anyway.
‘Attribution’ ain’t just a river in Egypt

CC-BY-SA3 requires (this is heavily edited for readability) If You Distribute, or Publicly Perform the Work or any Adaptations or Collections, You must keep intact the name of the Original Author (or username, if that’s what was used) for attribution.

That means that (for wikis,) every author who’s ever edited an article must remain credited forever. On a major popular Wikipedia page that might mean thousands of editors are legally required to be credited if that article is sourced, adapted or excerpted in any way. In short, an attached list of authors even longer than the license text GFDL requires.
This is about the point at which you should be realizing that CC-BY-SA3 is no more ‘intended for wikis’ than GFDL was. It was written for individual artists remixing one another’s work.

GFDL, by contrast, requires that re-users: List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

Much saner. …until you realize that wikis don’t have Title Pages, (Title Pages are one of those ‘invariant sections’ Wikis lack.) Instead author credit is handled under a sort of sideways interpretation of sub-section I. (edited for clarity) Preserve the section “History” and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version. If there is no section Entitled “History” in the Document, create one stating the title, year, authors, and publisher …

This is actually what TfWiki.net ran into when leaving Wikia… Wikia was inserting a link back to itself in the footer of every document, claiming that this was a legally required ‘author credit.’ That’s foolish on its face– it’s a publisher credit. TFWiki.net ran a full history import– which included a record of every single edit (with usernames) to meet this requirement instead.

This also means that the standard mediaWiki ‘article history export’ (which IIRC only exports the last… 200 edits?) exporter is technically in violation of GFDL, which requires that such credits be made easily available in a machine-readable form. Hrm.
This is about the point at which you should be realizing that neither GFDL nor CC-BY-SA3 was written with Wikis in mind.

GFDL tries to play nice with it’s Title Section allowing for primary author credits… but wikis don’t use ‘em, so everything defaults to the “history” section– which was never really designed for the purpose.

In March 2009, TFWiki.net suffered a database crash that wiped out about 7 months of edits. The articles themselves were recovered, but the edit history (who did what) for those months were lost… which technically places us in violation of section I– the history has not been preserved, though there has been a good-faith attempt to do so.

Here’s the thing– Wikipedia has billed itself as ‘the encyclopedia anyone can edit’ that anyone can use… but for the last 8 years, the GFDL terms-of-reuse technically required that anyone porting a copy of the article ‘preserve the history,’ of that article, which means a full-history export. Wikipedia has never offered full-history exports. This means that Wikipedia should technically have lost the right to use its own content under GFDL… but the reality is that everyone has simply been ignoring the issue as hard as possible.

The credit-preservation requirements under CC-BY-SA3 equally absolute, but dump the ‘preserve the history’ aspect that’s such a nightmare for wikis. (Full-history exports are about 20 times larger than current-state exports.) Ultimately, the Creative Commons license section 8, sub-section C includes the magic phrase that really applies here; If any provision of this License is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this License.

Author credit really needs to be treated differently on Massive Multiauthor Collaboration sites like wikis. CC-BY-SA3 doesn’t do that… but it does contain the magic language that allows you to ignore the problem legally while making a good-faith attempt to preserve the spirit.
In Perpetuity, throughout the Universe…

Both GFDL and CC-BY-SA3 grant the licensee ‘world-wide’ (or worldwide) rights to re-use the content on a royalty-free basis. That language really needs to be updated; right now copyright ends at orbit, but as the commercial exploitation of space continues to grow, the “legal wild west” enjoyed by the space program will eventually settle down. It’d be nice if the next version of GFDL and Creative Commons got ahead-of-the-curve and extended their language to the Solar System to prevent legal shenanigans.

What is a Derived Work?

So, aside from a few piddling specifics about whether or not a license is attached and just how authors are credited… GFDL and CC-BY-SA3 are substantially identical. They both allow you to make copies or modify content as long as the new work is available to the public under the same terms.

GFDL phrases this as a terse permitting of “modification or translation” of the original material. It was written for Software Documentation after all…. It’s not like someone’s gonna stage an interpretative dance of it. Creative Commons, written for artists, is more verbose;

“Adaptation” means a work based upon the Work, or upon the Work and other pre-existing works, such as a translation, adaptation, derivative work, arrangement of music or other alterations of a literary or artistic work, or phonogram or performance and includes cinematographic adaptations or any other form in which the Work may be recast, transformed, or adapted including in any form recognizably derived from the original, except that a work that constitutes a Collection will not be considered an Adaptation for the purpose of this License. For the avoidance of doubt, where the Work is a musical work, performance or phonogram, the synchronization of the Work in timed-relation with a moving image (”synching”) will be considered an Adaptation for the purpose of this License.

This is standard legal boilerplate. You can’t turn a book into a motion picture without paying the author.
But what about writing a sequel to a book? That’s not a re-casting of the original story… it’s an entirely new story told in a pre-existing setting. That’s something else entirely– surely it’s a derivative work, which the section goes out of its way to include?

U.S.C. § 101:A “derivative work” is a work based upon one or more preexisting works, such as a translation, musical arrangement, dramatization, fictionalization, motion picture version, sound recording, art reproduction, abridgment, condensation, or any other form in which a work may be recast, transformed, or adapted. A work consisting of editorial revisions, annotations, elaborations, or other modifications which, as a whole, represent an original work of authorship, is a “derivative work”.

That’s odd, US law seems to draw the line at abridged versions. A sequel to a movie would only be considered a derived work if it was nothing more than a re-arrangement of the events of the original movie with little or no original story added! (Insert your own joke.)

Under this logic, a new story told with someone else’s characters (an unauthorized sequel to Sleepless in Seattle, for example) would not be a copyright violation! It would be fair use!

Everything is a Derived Work

A combination of trademark and an expansive interpretation of copyright have combined to quash almost all most all such ‘unauthorized sequels’ told with someone else’s characters. The operative logic basically goes something like this:
It is impossible to separate a character from the story in which they first appeared.

There’s a certain validity to this argument. While every Spider-man story told since 1962 hasn’t involved him beating up muggers, virtually all of them have hung on re-treading his original guilt over his uncle’s death, his obligation to his Aunt May, or “With Great Power Comes Great Responsibility.” The character is the story, even if few characters are as clear-cut examples of this as Spider-man.

When you get to visual works, a similarly expansive standard has been applied. A copyrighted photograph reproduced as a background element on somone’s wall in a movie, barely glimpsed, is a copyright violation. Or, notoriously, a few seconds of the Simpsons visible on a TV in the background of an interview.

Fair Use is an unproven assertion

Fair use is: “not an infringement of copyright” under U.S. law. This is legally distinction from the way the rest of the world handles it; ‘fair dealing’ is a violation of copyright, but the manner in which it was committed provides a defense which justifies the violation.

On paper that sounds like Fair use is much better justified in the U.S.– no crime occurs here, wheres in the rest of the world a crime does occur, you just have a defense for it.
In practice, U.S. law has been distorted into a fair-dealing type system because it is quite common to have fair use challenged in court (or threatened to be challenged) as a copyright violation, even for uses that clearly fall under the doctrine. Fair Use is actually an assertion– no usage can actually be assured to be safe under fair use unless/until found ot be so in court– after being challenged.

The reality is that ‘fair use’ which is supposed to protect certain use of copyrighted material has been eroded into virtual non-existence. Any use of copyrighted material, must be legally accounted for.

Downstream Fruit of a Poisoned Tree

That’s all nice, but how does it affect Wikipedia?

In 2008, US presidential candidate John McCain gave a speech about the nation of Georgia which lifted several paragraphs, almost verbatim, from Wikipedia’s article on the country. He did not follow this with a 7 minute reading of the GFDL license. (Hilarious as that might have been.)

So was this Fair Use, or was the section cited enough to make the entire speech GFDL? Well Fair Use is a legal justification for using someone else’s work, but the uncited use of blocks of text from another work, even with the minimal paraphrasing seen here, is plagiarism. With no sources cited, let’s just call this ‘theft’ (it’d be regarded so in academia) and say that Fair Use doesn’t enter into it.

So the senator has mixed chunks of a GFDL source into his speech. If this were a proper relicensing then the speech (and his performance of it) would become GFDL. But the senator didn’t include a GFDL notice. Lets look at GFDL section 9, shall we? (edited for clarity)
You may not copy, modify, sublicense, or distribute [the Wikipedia article on Georgia] except as expressly provided under this License.* Any attempt otherwise to copy, modify, sublicense, or distribute it is void, and will automatically terminate your rights under this License, but does not terminate the licenses of parties who have received copies or rights from you under this License.

See that asterisk in there? That’s my addition. ‘As expressly provided’ here means in accordance with all the requirements of the GFDL… such as the requirement that a modified work also be GFDL, and include the full text of the GFDL license. The real critical section is that second part– McCain’s failure to cite and provide a license does not in any way prevent the work from being GFDL, nor does it stop anyone from re-licensing it.

Not a big deal, right? So the speech is GFDL, it’s a speech… he wants to get it out there!

So what happens when the speech is re-used? We’ll ignore news programs for the moment (since there are broad protections for fair use when documenting current events) and say that, for example… the speech is included in a later documentary about the 2008 election. That documentary becomes GFDL, even if the film-makers were unaware of it.
An unrelated interview from that documentary which is later re-cut for a Motion Picture to show the senator reacting to news that space aliens are invading… causes that movie to become GFDL. Anyone can make a copy, legally, or remix this Hollywood Blockbuster to their heart’s content.

That all of this stems from a documentary failure on the part of some speechwriter is irrelevant. If it was a normal copyright violation, you’d call this ‘downstream liability.’ In 1995 a piece of public art which had been donated to the city of Los Angeles, but whose copyright was still owned by the author made it into the production of Batman Forever. And by ‘made it into’ we mean “had a giant set reproduction made, and was featured in the trailer, poster, mugs, t-shirts…” Oops! The fact that the source of this screw-up was the City of Los Angeles not realizing that they couldn’t give away the rights to this sculpture in no way prevented the artist from seeking damages directly from the t-shirt manufacturers for every single violation.

But under GFDL, instead of cash damages, the ‘downstream liability’ is everything downstream becoming GFDL, something anyone can make a copy of.

CC-BY-SA3 works in an identical manner.

Giving Notice

Of course, all this could be avoided if McCain’s re-use of the GFDL material had been fair use– a citation of the source it came from and a statement that ‘this is fair use.’ TFWiki.net, which obviously contains a lot of images from Transformers, has to do this for every image on our site.

It’s not that claiming Fair Use necessarily prevents a re-usage from being a copyright violation– too large or too ‘central’ a chunk of the original material and it still might be illegal especially depending how it’s used. But because fair use is an unproven assertion until it’s challenged in court– the ‘downstream liability’ would not occur until and unless it’s found not to be fair use in a court of law.

A proper citation of fair use is like a legal dam; liability cannot flow into the derived works while the dam still stands.

The Hell You SA3

Remember back when I said that part of the Creative Commons GUI don’t actually match up with the underlying legalese, and it’d be important later?

The Creative Commons GUI claims that the right “to Remix — to adapt the work.” is governed by its license.

And remember when we discussed how derived works are limited to re-castings of the original work? Integrating parts of copyrighted works into a new work isn’t (or at least should not be) a copyright violations at all.

The Creative Commons GUI claims the most expansive possible definition of derived work– the kind of ‘everything is ours forever’ approach that the Free Culture Movement is always complaining has gutted the public domain and had a stifling, chilling effect on artistic free expression.

And remember that the GFDL actually claimed the narrowest possible definition of a derived or modified version? It was written to govern software documentation– it didn’t much care about preventing someone from staging “LISP Build: An Interpretive Dance.”

So now you’ve got two ostensibly identical licenses that fall on completely opposite ends of a broad spectrum of what can be done to original material under fair use. GFDL allows virtually everything that’s not related to documenting software, Creative Commons allows virtually nothing– any work derived in any way must still be licensed under CC-BY-SA3.

It may now be occurring to you that the ‘fruit of the poisoned tree’ argument I made regarding John McCain’s speech doesn’t actually work– he stole from GFDL Wikipedia, and GFDL reserves a very narrow definition of applications as also requiring a GFDL license.

But it would work under a Creative Commons license, which uses the broadest possible definition.

Wikipedia switched to the Creative Commons CC-BY-SA3 license on June 15, 2009.

Slow Pressure Over Time

Picture a world where Wikipedia, the most-referenced and least-cited resource on Planet Earth is infected with a disease. And using the content from Wikipedia without proper legal protection (citation and a claim of fair use.) infects any new work it was based on, it’s now a carrier.

CC-BY-SA3 diffuses slowly over time. Blog entries which draw on the site become CC-BY-SA3. From there it might jump into news articles– or art pieces (I like to put original art on my blog entries, and that art would become CC-BY-SA3. Imprudent use of Google Image Search would infect all manner of things.) Three or four generations out, its origins entirely laundered, it’s no longer possible to tell what’s CC-BY-SA3. Failure to note that a blog entry is CC-BY-SA3 doesn’t stop the re-usage of others from being CC-BY-SA3, because failing to abide by the terms of the license doesn’t prevent you from being a carrier.

The production department of a TV movie uses a graphic they got off the web for a sign. The image was CC-BY-SA3. By the expansive interpretation of copyright law that’s been so popular in the last few decades… the movie is now CC-BY-SA3. Anyone can make a copy, freely and legally.

Unrealistic? Maybe not as much as you’d think. Superstar comic book artist Greg Land is notorious for tracing images he finds on Google. Normally this is a minor copyright concern– but if he traces an image from someone’s FLCKR account that was licensed CC-SA3, the entire issue could become CC-SA3. Legal test cases involving Creative Commons indicate that courts will uphold this– and if Land’s employers are unhappy about the fact anyone can now make a copy of their comic, they can sue him for damages.

But what it mostly does is create massive ambiguity. Was Land’s use of the FLCKR photo fair use? Maybe, maybe not. But unless the issue included a fair use claim, there is now a reasonable basis to claim that the issue is now part of the Creative Commons.
And reasonable basises are what safe harbor laws are made of. It took half a decade and a supreme court to win a single legal judgement against The Pirate Bay even when a vast majority of the content it served were blatant copyright violations.
When comic books, movies and episodes of TV show begin to fall into a limbo of “could be available under the Creative Commons,” the burden is now on the alleged copyright holder to prove that the work isn’t creative commons before they can demand it be pulled. (Well, they can demand it all they want… but a site like that now has a genuine legal basis for ignoring them.)

The burden of proof shifts from a situation where individuals have to include disclaimers of fair use (and even then face possible legal reprisals) to one where the large media companies will have to include disclaimers of fair use, trying to ward off their own works becoming infected by the increasingly-omnipresent CC-BY-SA3, which is like the public domain– except that you can’t use it without also being infected by it, becoming this new sort of pseudo-public-domain where anyone can copy.

By 2030, either almost everything published everywhere in the world will be CC-BY-SA3, or we’ll be heading for a series of worldwide Supreme Court showdowns as the same large media companies that spent decades claiming that they owned everything discover that that same expansive interpretation of derivative work they championed may now cripple their ability to own anything.

Cui Bono?

This whole argument is made of sticky-tape and dreams… just like the rest of our legal system. It’s a brief; a possible scenario. But test cases involving the Creative Commons (mostly around FLCKR accounts) seem to indicate that this basic premise is legally valid, and the text of the licenses… says what it says.

It’s possible that the world will go on functioning with no problems whatsoever. The infectious nature of the CC-SA3 license will be ignored, just like everyone ignored how Wikipedia wasn’t properly compliant with the GFDL’s attribution clauses.

The real question you might want to ask yourself then is why there was such a big push for Wikipedia to change licenses? Because CC-BY-SA3 is a slightly better license, still horribly unsuited for wikis? Because getting every major source online to use Creative Commons allows for portability of content? It doesn’t. CC-BY-SA3 content cannot be ported to a CC-SA3, CC-SA-NC or CC-BY project… each license is it’s own thing, legally, they just share a common umbrella name. Is it because the ‘must include the full text’ proviso that everyone ignored was too cumbersome?

This is a video of the November 30, 2007 iSummit in San Fransisco where Jimbo Whales announced Wikipedia had brokered an agreement for the switchover.

Why does the room go up when Jimbo makes his announcement? Jubilant shouts of “we nailed it?” Why does Lawrence Lessig hug Jimbo Whales and say this announcement is the greatest event in his life after the birth of his children?

This move was pushed for, hard, by the radical arm of the Free Culture movement. And while Wikipedia’s important and all… why is change from one piece of legalese to an ostensibly identical piece of legalese causing blissed-out joygasms from the people who most understand what these licenses mean, and are most vocal about the need for radical change in how copyright is handled?

The real question is what do they see that has them so excited? Is it the year 2030?

"Invite them in, let's take them in. Let's be like the diseased prion that destroys its host in CJED. Let's go in there and give them something they cannot digest, something they cannot process. Something so toxic, so dangerous, so powerful that it will breed and destroy them utterly. Not destroy them-- turn them into us! Because that's all we want!" Grant Morrison at Disinfo.Con 2000, excerpted as fair use for academic discussion

This blog entry is released under CC-BY-SA3.

Whos who at the Laurentian Abyss

2009-06-27T02:09:04Z

Transformers; Revenge of the Fallen has dropped.

I saw it on Thursday. It really doesn’t hold together as well as the first one. Sub-plots and character scenes got cut left-and-right int he second half, and the result is that it’s almost solid robots, and Epps and Lennox feel a little useless. It’s a bit frustrating because review after review talks about being baffled by the plot, and I know it’s supposed to make more sense than that… it’s jsut that about half the movie’s exposition ended up on the cutting room floor. Bummers.
On the plus side, stuff explodes like every 30 seconds!

Not helping with the confusion in the film is the bazillion robots in the film. The official total is 42, but I’ve heard some people who were counting say that total was much higher.
Since they’re often in motion (stomping around like bags of jello with metal parts suspended in them) and almost none of them are named, telling who’s who can be a bit… challenging.

Fortunately, I have resourceful friends. So I present screencaps!

The Laurentian Abyss sequence has 4 Decepticons (plus ravage) diving off a ship.

Then then jump off the boat. First a tan fellow with long claws, then Mixmaster with his shell-wing things, then Long Haul (the hulking green one) and a red version of the pogo-stick-bot (Rampage or possibly Skypjack) does a backflip in front of Longhaul, actually entering the water ahead of him.

Later, two of the Transformers turn on a third. This outline could conceivably be the Jackhammer-bot, but he’s actually offscreen to the left… so it’s got to be the long-armed Decepticon who exited first.
And the clamp on his arm (seen earlier in vehicle mode) makes the victim readily identifiable as a Constructicon Scout.

Unrelated note: David (the inveterate movie pirate who got these to me) tells me the ROTF bootlegs out at the moment are “all crap,” and points out that he went to see ROTF in theaters.
(He recommends the IMAX experience.)

10 of the Most Ridiculous Transformers, Ever

2009-06-24T02:15:35Z

The Street Level blog recently posted a list of the “Top 10 Most Ridiculous Transformers, Ever.” Their choices were boring– all from the original cartoon and not very ridiculous at all.

As a Transformers nerd, I know exactly how ridiculous Transformers can get.

01Name: Piano Transformer guy (no formal name)
From:Generation 1 Marvel UK comics “Target 2006”
Year: 1986
The meek “Piano Transformer guy” transforms into a Player Piano in Maccadam’s_Old Oil House, a time-and-dimension-spanning bar on Cybertron. Maccadam’s is recognized as neutral ground by Autobots and Decepticons, and both sides rub shoulders and knock a few back. Here the Piano Transformer guy runs afoul of a foul-tempered Decepticon with no taste in music.

02Name: Esmeryl
From:Japanese Transformers Victory manga “Heroic! The Victory War”
Year: 1990
Esmeryl is the wife of the thoroughly badass and yawn-inducingly boring latter-day Decepticon leader Deathsaurus. At the end of the Victory series, just as the Autobots had defeated the Decepticons and were about to destroy them, Esmeryl appeared and begged for the Autobots to spare them– revealing that Decepticons only stole energy to feed their families, cue the appearance of hundreds of adorable Decepticon children crowding around the ones about to be executed. (Needless to say, the Autobots caved.)
Esmeryl, like her husband, appears to transform into a dragon-like Kaiju. Esmeryl is apparently sterile, having no children of her own. Instead, she and her husband adopted a human boy and turned him into a cyborg. Awww…

03Name: Misfire
From:Generation 1
Year: 1988
Misfire is a Decepticon with no aim. He couldn’t hit the broad side of a barn from the inside of a barn. Repeatedly flunked out of Decepticon Military College, only granted an honorary graduation after the college itself was destroyed by Autobots.
Misfire is a Targetmaster, partnered with a smaller robot that transforms into his weapon and aims for him, freeing him up to concentrate on other battlefield tasks. Unfortunately, his partner goes by the codename “Aimless“…
Whichever Decepticon human resources assigned them as partners should really be fired.

04Name: Pepsi Convoy
From:Generation 1, Japanese toy-exclusive
Year: 2005
There was a “Pepsi Optimus Prime” in the United States as well, but here it’s just Optimus Prime disguised as a Pepsi delivery truck. In Japan he’s a completely separate character.
Product of the same NASA experiments that created the Japanese superhero Pepsiman, Pepsi Convoy is a tiny robot who live in Optimus Prime’s freezer and battles the forces of an evil Decepticon-owned pizza chain with the power of his CO2 gun.

05Name: Sky-Byte
From: Robots in Disguise
Year: 2001
Sky-Bite is Megatron’s right-hand-’bot, a fearsome figure as he cuts through the air in his atlmode of… a singing shark?
Apparently Sky-Bite used to be quite fearsome, but he took one to the head in his first appearance, and ever since then he’s been a bit… off. A vain, preening insecure prima-donna who write Haiku’s to his own accomplishments… but is devastated when Optimus Prime criticizes the derivative imagery and non-cohesive structure.
Sky-Bite is pretty much pure liquid awesome, even in a series like Robots in Disguise (which was a comedy.)
How can you not like a guy who sings his own theme song?

06Name: Legion
From: Kiss Players
Year: 2006
At the opposite end of the comical-to-creepy scale from Sky-Byte is the Legion, an army of penis-tongue’d Megatron-lookalike Decepticons created when Galvatron crash-landed in Japan and his immune system went nuts. The Legion exist to rape (or possibly eat) little girls and use them fashion horrible new bodies for themselves.
Opposing the Legion are the government-sponsored Kiss Players, a group of abused girls who give Transformers special powers by kissing them. Nudity, fetish-vomiting, panty shots and a human-on-transformer rimjob follow in short order.
Bravo Japan, way to avoid Western stereotypes about your country!

07Name: Drift
From: IDW’s modern Generation 1 comics
Year: 2008
When IDW relaunched their Transformers comics All Hail Megatron, they decided to really shake things up by introducing a new character; Drift.
There’s nothing wrong with that, Transformers adds new characters on a regular basis. But Drift is that special “designed-by-committie awesome” quality; he’s an albino-japanese-streetracer-taciturn-former-Decepticon-honorbound-samurai-ninja.
How awesome is Drift? Look at that picture? He has three swords. Three at once!
Drift’s actually been pretty inoffensive in the actual stories he’s appeared in, but any character IDW Comics refers to as “our Wolverine” during interviews has a real uphill climb towards acceptance.

08Name: Break
From: Beast Wars Neo
Year: 1999
Break is a insubordinate foul-mouthed snot-nosed pubescent ‘git who turns into a penguin.
When his injured commander Big Convoy made a rare confession that he was proud of his crew, Break replied by insulting the size of his penis.
He also sounds exactly like Naruto, since he has the same voice actor, doing the same “type” of voice for both characters.
Basically: “Penguin Antagonist Naruto” sums up Break pretty nicely.

09Name: Shortround
From: Transformers Cybertron
Year: 2006
Shortround is a Decepticon nerd. Now, Decepticons have a certain tolerance for nerds when they’re using their nerd brains to figure out how to steal cable or design esoteric doomsday weapons… but Shortround collects toys. Specifically rare, convention-exclusive, limited-edition toys, which he’ll often wander off his mission parameters in search of.
Between his giant clumsy pincer hands, his unrequited crush on his sexy nipple-flashing teammate Thunderblast, and his tendency to vomit napalm when nervous, Shortround makes a surprisingly touching metaphor for adolescence.

Ah… my teenage years. I remember the first time I vomited napalm…

10Name: Signal Lancer
From: Transformers Cybertron
Year: 2005
Signal Lancer isn’t much of a figher– he’s a civilian. So when the entire population of Cybertron evacuated to Earth and took disguises, he chose a nice safe altmode; a stoplight. Signal Lancer was a stop-and-go running gag throughout the Transformers Cybertron series, but he proved himself surprisingly useful in a pinch and when the series ended he became a cartographer, mapping Earth and the galaxy.

So there you have it. 10 ridiculous and odd Transformers. This barely scratches the surface. There’s more than one Transformer that turns into a toaster after all, and no one can forget The King, no matter how hard they try…
I’d like to thank TFWiki.net for suggesting some of the more obscure ‘bots on this list.

TFWiki.net: serious intellectual debate about transforming space robots

Orion Bursts His Belt

2009-06-10T20:44:31Z

I’m always on the lookout for news stories that sound like the first act of disaster movies or cautionary tales. Most of them turn out to be nothing (”super-cures from Alligator blood!”) but they’re fun to keep an eye on.

One of our largest stellar neighbors, Betelgeuse is mysteriously shrinking, to a significant degree (it’s lost 1/6 of its size.) No one knows why, but they’re not very concerned about it either.

Now, I happen to be aware that a supernova sterilizes the entire ’stellar neightborhood’ for many lightyears around, so I experience some normal concern when I hear that part of Orion’s belt might collapse or explode.

Here’s the numbers– Leo F. Rog posted a generalized summary (based on old sci.astro consensus) of what would happen if a nearbye star went up on “Bad Astronomy,” a blog which debunks Hollywood depictions of space.
The upshot is that Betelgeuse is ~500 light years from Earth, which puts us out of range of everything but the gamma radiation burst– which would be many thousands of times that amount of Gamma radiation that Earth sees during a solar flare. No one’s quite sure what that would mean, but gamma radiation is something our atmosphere blocks pretty well.

This sounds unlikely to wipe out all life on earth even in a worst-case scenario. I’m pleased!

Three things I have learned about AS3 and Flash recently

2009-05-29T17:10:30Z

I’ve been developing a Flash application for a CD-rom recently for my last internship before graduation. Permissions, priorities, and even some of the design aesthetic are… totally Bizarro-world compared to web applications. It’s like looking at the tool from a completely different angle.
Here’s some random thoughts/discoveries from the last few days:

AS3’s TextField CSS rules don’t really support contextual inheritance, and it fails badly.; .credits p {} not only doesn’t work– it will cause The rest of the Stylesheet it’s not included in to not work. Naughty naughty!
Importing video with alpha is surprisingly painless. (v9+)
Resize-with-player is both extremely simple, but not as robust as you might expect. (It’s totally globalized… but you wish it wasn’t.)

Nothing groundbreaking, just there.

Citation Needed - Linguistic Akido You Need To Know

2009-04-13T08:38:29Z

Language is a funny thing. Humans used to believe that words had power– that monsters cannot cross a threshold without being invited in, or could not deny requests phrased in a certain way.

I have this fantasy

The note [citation needed] is found all over Wikipedia. Whole articles are frequently tagged with the dreaded “this article does not cite any sources” notice. It’s not that anyone necessarily fears that the information presented is incorrect per-se… but providing a source;

Allows Wikipedia to say “Well it’s not our fault we got it wrong.”
Filters out some of the cranks by requiring they find a corroborating source.
Gives the reader a starting point for seeking more information.

Over time citations, should increase the reliability or articles via slow pressure. (Even if that timeframe is geologic.)
In practical terms, Wikipedia sometimes runs into Strange Loop problems, like the story about the musician who added a few sentences about his father to his Wikipedia bio (Note: No one should ever edit their own Wikipedia bio.) and was confounded to see a “[citation needed]” appear after his additions because it was information that had never appeared in a book or interview. It was true, he knows it’s true… but how do you cite something like that on a website with a No Original Research policy forbidding Wikipedia from being its own source?
The problem fixed itself a few months later when a newspaper article which included information on his father as background. It could now be cited. The uneasy thing about this resolution was… the newspaper article almost certainly drew that information from the Wikipedia article. The repetition of an uncited source has, in effect, “laundered” the information, it is now citable– no more verifiable than it ever was; it’s ‘factiness’ has simply reinforced itself. A false statement that the artist “loved black licorice” might become “citable fact” in exactly the same manner.

A lot of people seem to feel that this is a problem with Wikipedia. I argue that it’s actually a problem with everyone else but Wikipedia. If the newspaper had provided a source, or even a bibliography, it would be readily apparent if it was referencing the unverified Wikipedia article.
When politicians throw out a crazy number like “70% of all households that…” wouldn’t you love to be able to instantly check and see where that figure was coming from? To find out what kind of standard was used to arrive at that terribly impressive number?

Sad but true: Wikipedia has a higher standard of proof than network news.

Stephen Colbert is one of the masters of “magic language” in America today… right now it looks like that’s going to get a node on the International Space Station named after him. And last Tuesday it got him knighted by Queen Noor of Jordan.

Naturally, Wikipedia demands citation. Not of the knighting itself– that got enough press attention– but for the modification this imposed of vainglorious title Colbert’s character uses to refer to himself by. Perfectly natural, there was much argument on wikipedia about how to properly register his honorary doctorate. (The common assumption, “Dr. Stephen Colbert” being incorrect.)

So, for the record…

The April 7 credits; 'Sir Dr. Stephen T. Colbert, D.F.A.'

On the April 7, 2009 telecast of The Colbert Report, Queen Noor of Jordan knighted Mr. Colbert, as “Sir Stephen, leader of the Colbert Nation.” Mr. Colbert’s Executive Producer credit was modified to read “Sir Dr. Stephen T. Colbert, D.F.A.,” beginning with that same show.

Linguistic forms fulfilled, this fact is now citable. A blog isn’t the best quality source, but for a simple matter of fact it should be more than sufficient, especially when accompanied by pictorial evidence.

The other “magic” linguistic trick used in this blog entry involves copyright. To use that screengrab, I have credited it thusly;
Image (c) 2009 Comedy Partners. Excerpted here as fair use for reasons of citation and commentary.</div>

Credit, unambiguous statement of non-ownership, a point in the direction of the owners, and a simple statement of why I get to use an image that belongs to someone else.
It’s not really that dissimilar to the logic behind providing citation on Wikipedia, is it?

Everyone should know how to properly credit copyrighted material for use. It is a necessary skill in an age when the Public Domain has been so badly eroded and everything around us is owned by someone else. This protects you, and it protects them.

Some people are confused when I say this, since I sometimes write about piracy in my blog… but while I think that Copyright law in the US has been expanded beyond all good sense, I am not in favor of piracy per-se. I just find piracy interesting from a force-gradient perspective.

Fox is trying to give away most of their shows for free on their website already after all, it’s just that the distribution channel they’ve chosen for this is so bad that most people would rather break the law to watch those shows online rather than do so legally through Fox’s site.
Fox feels this is a problem with the rest of the internet, rather than their site.

I feel like I should care about that distinction more, but right now I’m more interested whether Jordan now recognizes Stephen Colbert as a Head of State. Queen Noor did call him the “Leader of the Colbert Nation” when she invested him after all…

Man vs. Internet

2009-04-05T20:47:45Z

I haven’t watched television since about… 2002? I gave up on normal TV shows first, news was the last to go. This gave me a lot of free time since in 2002 television hadn’t yet migrated online. That’s reduced as I started guiltily following TV shows again (starting with House,) but still nowhere near the 4+ hours a day the average American spends watching TV.
I’ve also come to know the pain of having a favorite series canceled (I think I’m the only one who watched Blade), and my current favored pleasure is Terminator: The Sarah Conner Chronicles.

Terminator has at least half a million people downloading it every week. (I say “at least” because direct-download is growing, and that doesn’t show up on those numbers.) It’s also been canceled, which strikes me as a really odd move with a movie months away. This makes me think that either;

The movie really, really stinks.
It’s a ‘fake’ (or at least tentative) cancellation to see what happens.

What I see, trying to grab the latest Terminator episode, is that almost every link to it was dead within 2 hours. That the ones which weren’t were dead were bogus (containing deliberately corrupt archives or only the first 15 minutes of the program) and foreign video-sharing sites were having spastic twitchy meltdowns I associate with denial-of-service attacks (or possibly just massive overload) and were unviewable. And meanwhile, on SurfTheChannel (a watch-TV-online indexer site) all the links are dead or nonfunctional except “buy on iTunes” or “Watch on Fox.com.” (Note: This was last night at peak usage. 24 hours later Youku appears to be sorta working again.)
I have no ideological problem with watching the episode on Fox.com where they feed me fewer commercials, but ones targeted to my demographic information. (I doubt Fox.com’s ability to stream half a million copies of the episode as they do their damnedest to cripple the distribution network sending pirated copies around the world, but that’s a separate issue.)
And then I get there– and I remember why I don’t use Fox.com. They want to install software on my computer before they’ll let me view the episode. No.
I repeat, with emphasis, NO. All these sites Fox appears to be going out of its way to strangle in order to force me to use theirs are able to deliver fine quality video without first demanding that I install 3rd party software on my machine. So may I offer a giant flaming “No and Fuck You Too” to Fox.com’s kind offer.
I go to sleep. And I wake up to this:

Swedish traffic halves, no more pirating

Posted on 05.04.2009 at 11:31 in
Tech News by Martin

Internet traffic in Sweden has plummeted after a tough new anti-piracy law was enacted in the country earlier this week, casting interesting new light on the extent to which illegal file-sharing occurs. The new law makes copyright holders such as record and entertainment companies to go through the courts to determine the identities of those suspected of piracy, via their IP addresses. The anonymity illegal file sharers have has hitherto made the practice widespread, although figures vary as to exactly how commonplace it is in various countries. However, traffic to Netnod Internet Exchange AB, a Swedish firm which manages many of the country’s key internet exchanges reported a drop of around half since Wednesday, when the law took effect. Throughput has yet to pick up.

Data transmission rates have slumped from a peak of around 190/200 Gbit/sec to daily highs since Wednesday of about 100 Gbit/sec. At the time of writing, the figure was around 80Gbit/sec. Along with its Scandinavian neighbours, Sweden has one of the most mature internet industries in the world, with a highly developed fibre-optic network broadband infrastructure. The figures will be a shock to many, pointing as they do to a potentially high prevalence of illegal file sharing. France yesterday showed its commitment to eradicate illegal file-sharing after passing a “three strikes law” which decrees that persistent offenders can be suspended from using the internet for a period of time. However, moves to get the law enacted at a European level have met with more opposition.

Source: Vnunet

Well… we knew it was coming.
Ballparking some math… half a million copies of T:TSCC at 300mb+ each is about 130 terabytes, and divided over the 24 hours of peak downloading… that’s about 13GBit/s, or about 17% of the missing internet traffic solely attributable to Fox’s actions. (I’m being semi-facetious here.)
Regardless, with Fox flexing its muscles and Sweeden setting sweeping new legal precedents, now is definitely a time to keep one’s head down. If Fox is trout-slapping pirate distribution channels, BitTorrent (where everyone can see your IP address) is probably not a good idea. Say… trout-slapping, that gives me an idea!
So I find myself on Efnet, a part of the internet that predates the World Wide Web (it’s first major internal schism occurred four years before web browsers were even invented) that’s remained essentially untouched while the internet evolved around it. Efnet in specific was founded by the modern cult that worships the Greek goddess of chaos, which i appreciate.
FServ’s are a type of direct client-to-client file server that leave no records, log no IP’s, don’t occur in a web browser, cannot be listened in on, and are essentially 100% untraceable because they occur between two individuals– no vast pool of seeds like on bitTorrent, no central download server like on Rapidshare. They are less convenient than modern file-sharing methods… but much much more secure. (And in most cases more searchable.) They have no backbone or infrastructure to be torn down or attacked, they can move with 5 minute’s notice– and any legal attempt to monitor their gross traffic would involve exposing thousands of private conversations completely unrelated to the file sharing that isn’t even (properly speaking) occurring on the network– the transaction is entirely between the two individuals. As private as a streetcorner drug deal. Note to self: Find better analogy for this.

Ooh, bonus! I’m getting a faster transfer rate than I did on Rapidshare! Thanks for pushing me to another distribution network Fox!

Obama’s college writing under scrutiny

2009-04-01T19:46:31Z

Marsailles, France (VP) April 2,2009 — President Obama suffered another setback today as excerpts of an erotic fan-fiction written while in college surfaced on the internet. (Full Story)

http://tc.indymedia.org/2009/apr/obamas-college-writing-under-scrutiny

Five Servos of Seperation - Kevin Bacon Goes to China

2009-03-27T16:36:15Z

Continuing what appears to be a vague Transformers theme to my recent posts… for the second time in a week, the new episode of Transformers: Animated has “dropped early,” or as TFW2005’s news post so tactfully put it:
Once again, Cartoon Network and Turner Broadcasting have succeeded in leaking another upcoming episode of Transformers Animated.
Passive aggressiveness aside, as a lifelong Transformers geek, this is a bit distressing. The current TV series– widely regarded as the best thing Transformers has done in over a decade– has been relegated to Timeslot Hell by the network airing it. And why not? TF’s viewership is less “sticky” than other Cartoon Network series (tending to tune in just for this show) and while CN wants to target a demographic that skews older, the show is widely downloaded by that demographic since they have no desire to get up in the wee hours of the morning to catch new episodes.
Cue a positively-reinforcing cycle of downloading depressing ratings with results in a bad timeslot, which encourages more users to download.

Are you being served?

Cartoon Network allows users to view the episodes online via their website– a few weeks after their premiere, so as not to depress ratings. In theory this encourages viewers to watch the episodes when they’re broadcast instead of online.
In practice it really just make the online video service completely irrelevant, it is tasked itself to “reruns,” not premieres. Youtube doesn’t have an arbitrary imposed blackout period on new episodes, so people watch it there instead, an hour after they’re broadcast.

At least, that’s the normal state of affairs. Recently however you’ve been able to watch the episodes on Youtube before they’re broadcast. Cartoon Network uploads the episodes several days or weeks before their premiere, and simply doesn’t “hook them up.” By changing a few episode numbers by hand– viola!
No, actually, it’s even worse than that. Because you can’t watch the episodes through Cartoon Network’s player (which includes matrix to track viewership,) you can only download the files directly with no advertisements, and no way for CN to know someone is watching.

This is distressing from a security standpoint (the files should not be downloadable at all, let alone for yet-to-premiere episodes,) but more dishearteningly from a systems standpoint– because the CN video player backbone already has in place has a very simple mechanism built into it that would prevent this sort of thing… they simply choose not to use it. (I decompiled their player code last night to check, it’s there!)

Channels of Distribution

What does piracy mean in practical terms? When the episode dropped early I uploaded copies of the .flv files onto my own server for a friend in Canada, since the Cartoon Network site is geo-locked for United States visitors only. From there is gets murkier– a hidden copy of the link was posted on an internet message-board, when very few people seem to have noticed it. But analyzing my server logs, I can see a series of people accessing those files with no referring URL, which means they got the link instant-messaged to them from someone– like a game of “telephone.”
Two hours after I uploaded the files for one guy in Canada, a link to my server was posted on a Chinese message board, and there are suddenly have several Chinese Transformers fans downloading it from me. …Oops.
10 hours (and a signifigant chunk of my monthly bandwidth allotment) later, that’s trickled down to nothing. The episode finally made its way to Megaupload, and from there to Torrent sites (the reverse of the usual pattern!) where much vaster numbers of people will download them. I have no idea if the versions being uploaded came from my server, or via someone else independently exploiting the same security flaw. It’s about 50/50 odds either way– one badly secured file was “transformed” into 30,000+ downloads in the span of about 10 hours. The episode won’t air for another 24.

That’s slower than usual. Usually it only takes 3 hours.

Is There Life on Mars?

The music industry spent a decade fighting the idea of digital music before embracing it– with restrictive Digital Rights Management. They’re now slowly coming to terms with the fact they have to give that up too– because by the time they came around, the consumers had already embraced devices that made that way of doing things a restrictive nuisance.
Television fought the same battles– and is now finally grudgingly releasing their content on sites like Hulu, generally under the same “watch the episodes after a blackout period” system, kneecapping their content in the name of protectionaism for their broadcast advertizers.
And as-can-be-expected, because viewers want to see them NOW, a network has spring up for encoding and distributing the episodes hours after their air– “Fuck your blackout period.”
These pirated episodes, unlike the ones on Hulu.com, have no advertisements whatsoever. If Hulu offered the episodes the same time that they aired… there would be no pirate network. (Or at least it would be very small.)

So the question really becomes… how can the networks sell ad-space on a pirated television show when the people doing the piracy cut out ads?

I argue “yes,” but that’s a subject for another time.

The Host with the Least

2009-03-26T22:33:24Z

This is the new face of the Transformers wiki;

For an increasing number of hours per day, the site experiences network timeouts, database locks, service brownouts and straight-out “nothing there-ness.”
In the aftermath of our hosts accidentally deleting our database, the actual hosting TFWiki is receiving as we try to rebuild has actually gotten worse.

There is still no word on the server logs we need… Just a great big sucking silence coming from Texas.
For months they’ve done nothing– including clearing up our billing issues– and the one time they do do something for their own mysterious reasons, it shatters like a clay pot. And instead of springing into action to help clean up the mess they created– they revert back to a wall of silence and inaction while our site get more and more unstable.

I can’t call them “worst hosts ever,” I’m sure there are companies out there has done worse by their clients… possibly gotten them killed… but certainly the worst I’ve dealt with– and I’ve had a couple projects that went through hosting hell.

Urge to be nastily unprofessional… rising.

25,000 Pages of Pain

2009-03-23T20:48:20Z

The Transformers Wiki suffered a serious database fault on March 15th. It’s been 9 days.
We’ve had a tech-guy on standby try and reconstruct the database for 7 days. He’s a specialist, he does not come cheap, and he’s doing this on a volunteer basis. Our web-host is dragging its ankles, demanding teleconferences and approvals and all sorts of limp-wristed bullshit rather than give us what we need to try this.
Understand– no action is required on their part. All they have to do it give us access to some raw logs. Or dump us copies. And it’s taken them seven days, with no sign of action on the horizon yet.

I once had a web-host delete a dev site and database– including all back-ups– the only copy of this software we’d spent weeks developing. Totally their fault, they had a brain fart, thought “we didn’t want it anymore” and didn’t bother to back anything up before acting.
They at least had the grace to seem sorry about it.

I really get the impression that our host is simply dragging its feet, based on the logic that if we do enough recovery work– hand-remeshing 9 month-out-of-date files, and de-parsing HTML from internet cache’s for 25,000 pages… that it simply won’t be worth the effort for us to hold their feet to the fire– escaping the TFWiki’s righteous wrath simply because the diminishing returns offered by continued attempts to wring cooperation out of a them will make us lose interest.

That pretty much pisses me off. They may be content to waste our high-priced tech-guru’s time, and our time (he’s already done some preparatory coding for the files they show no signs of giving over,) but I value my own time. And I don’t take kindly to putting in more than a hundred hours trying to re-create work that our host lost due to a series of screw-ups. If there is not see some indication hour web-host has exerted themselves on our behalf, I am billing them for my time spent cleaning up their mistakes.

I was ballparking the amount of man-hours sunk into this recovery so far– and I came up with at least $30,000.00 worth of man-hours spent scrambling to recover from this disaster. 25,000 web-page caches tracked down and saved essentially by hand. Custom processing code written in Perl, Python and PHP to facilitate processing. Iteratively rolled-out tools developed in response to the community’s needs, dozens of hours spent scratch-rebuilding recursive templates– and the place still looks like a disaster area.

The reason for the unannounced upgrade that resulted in this disaster (we’re told) was a desire to reduce database load. The community hadn’t noticed anything, but apparently it was such a concern for the host that he went in and spontaneously upgraded our software- “unloading” the database in the process. Didn’t make a backup before hand, discovered that our nightly backup procedure (which had been knocking the entire site offline for an hour every night) was somehow misconfigured and not being saved. “Oops.” Four days after the crash, our host repeatedly reset and then totally killed the site because it was being “hammered” by someone– which happens to line up exactly with one of our users downloading the 1.7gb image backup. That transfer maxed out at 150kb/s. Our host’s response was to kill the site for 6 hours while it was “moved to another server–” yet the Wiki continues to bog down at the same time as the sites it used to share space with. VPS? Maybe, but then why were DNS entries changed?
We really have no idea what setup TFWiki is running on right now. We were told we had dedicated hardware and pipe- but it behaves for all the world like a virtual machine on a throttled connection. A miswired, badly-configured non-load-balancing connection.

The site has been growing more unstable over the last several says, database timeouts, unreachable network brownouts lasting several minutes occurring almost hourly. Part of this is probably due to traffic– all those people putting in time frantically trying to get the site up and running again. But it’s been a week, shouldn’t the host be able to adapt to this?

This whole thing stinks. We’re having smoke blown in our faces, and nothing feels right. I’m growing increasingly convinced that what we were sold when we signed up, and the actual set-up we were running on were two very, very different things. Most people only use a tiny fraction of their hosting capacity– we were sold the promise of a powerhouse web-host we could grow into, but after 9 months it’s feeling more and more like a 4 cylinder Geo Metro. And I’m no longer willing to assume in good faith that there’s a V-8 hiding in there, given the mounting evidence to the contrary.

The really disgusting thing is, we’d have installed our own backup extensions by now if we were given proper access. Despite 9 months of requests we were unable to get a FTP access set up for one of our people.

It is a measure of how pissed off I am that I am not mentioning our host’s name. The point is not to drag their name through the mud– that would be juvenile and pointless. They have to do business under that name. This is about me venting, not trashing them in a way that will lead back to them in Google. I have been disappointed on basically every level of professional conduct by these guys. While I can’t imagine any circumstance under which TFWiki would choose to stay with this host… it is my hope that they learn from this experience, and become a better hsoe for other people in the future.
I know who TFWiki was sharing our un-backed-up server with. Some of those people make their livelihoods by that content. The Wiki is at least mostly recoverable- with dozens of people putting literally thousands of hours into it- any individual losing all their stuff to this bungling wouldn’t have had that option! Let this be a learning experience for our host, so that this never happens again.

Remember the host I mentioned who once deleted a live dev-site with 3 weeks worth of our work? You better believe they learned from the mortification of that experience. I on the other hand, had learned the lesson beforehand I download a full mirror of the site every morning, and a backup of the database every 3 days because I don’t trust people.
Walking into work that morning and seeing the ashen faces of my co-workers as they explained everything was gone– then being able to tell them I had a complete backup on my desktop just out of force of habit? That is a precious memory.

We are all professionals here, let us act like it.
And as a professional, the 112+ hours I’ve clocked in so far cleaning up their mess has a cash value of $2240.

The Second Law of Thermodynamics and SQL

2009-03-17T08:30:20Z

This does not go in “reverse”

Late Monday TFWiki.net (the Transformers wiki) suffered a serious database fault in the midst of what should have been a routine software upgrade, completely wiping out about 9 months worth of data and fragging the user accounts.

Causes

The root cause of the fault remains obscure. It was accompanied by a spate of mass-vandalism. TFWiki attributed the problem to the “Bookworm Virus,” and a message was posted on an “underground hacker website” claiming responsibility for the hack.

Bookworm Teabags the Transformers Wiki

The actual chain of events is more ambiguous. While the database crash and subsequent vandalism may have external causes, complicating factors that rest squarely with the TFWiki host. He failed to make a database backup prior to upgrading the software, so when the database imploded, there was nothing to revert back to. The next obvious choice– revert to one of the regularly-performed backups– failed when an increasingly-frantic ransacking of the host’s archives revealed that there were no regularly-performed backups, the TFWiki server hadn’t been set up for them. Oops.
(This is perhaps more a matter of concern for members of the Blank Label Comics whose Webcomic archives and subscription databases are hosted on the same server and presumably also not being backed up.)
This follows the Transformers Wiki’s semi-tradition unfortunate timing; Wikia buggy Monaco “upgrade” was rolled out during the 2008 Transformers convention (while all the admins with the power to fix things where gone, but also a peak-traffic time,) they migrated to their new host (located in Houston) just as Hurricane Ike was causing rolling blackouts throughout Texas, and the Bookworm Virus struck right after the TV-movie that kicked off Transformers Animated’s final season.

Effects

“Yesterday is yesterday, if we try to recapture it, we will only lose tomorrow.”
–Some Guy
TFWiki is moving on. With the prospects of any sort of database backup turning up looking increasingly remote, TFWiki is digging in, scraping the Google cache to recover page text (a process that even with scripts helping will take between 2-6 days) and resigning themselves to reformatting by hand more than 8000 pages of raw html.
Myself, I’m weeping, because I’m resigning myself to reformatting the templates.
It would be one thing if the loss was total– then you could give up. But the job ahead– recovering 9 months worth of work by hand, is simply dishearteningly large. All the King’s Horses and all the King’s Men will put Humpty Dumpty back together again. …at a cost of time and effort roughly equivalent to the the original construction.
“In a system, a process that occurs will tend to increase the total entropy of the universe.”
–The Second Law of Thermodynamics
Order does not spontaneously arise from disorder. Ice cream doesn’t stay frozen in a hot room, piles of cards do not snap up into neat houses, and eggs do not spontaneously unscramble themselves.

And yet… while entropy will never decrease in a closed system, it can sometimes decrease locally, like the unscrambling egg this is merely phenomenally unlikely, not impossible.

Most of my friends call me a “tech-guy.” The dirty little secret is… every tech-guy has a tech-guy he calls for advice when he runs into something that’s out of his depth. Mine is Andrew Burton, he’s one of those guys who blogs about programming languages for fun. I wanted advice on customizing Warrick, a cache-scraper script written in perl. Like all good tech-guys Andy listens for about 5 minutes before veering off on a tangent and asking if we have access to the server’s SQL logs.

In the past Andy has, in desperate straits, had to re-create a deleted database without, well… a database, or a backup or export of any sort. This involves writing custom filters for the Server’s SQL activity logs and rebuilding it forward in time, piece by piece. Where a normal “page export with history” has dozens of revisions per page– this method has dozens of queries per revision. Reassembled in such a fashion, like building a human being from a map almost molecule by molecule, TFWiki’s 500,000 page revisions for 27,000 pages will translate into tens of millions of individual SQL queries. If every file is in place and intact, and every filter properly coded… this method can be used to grow a new functioning database architecture, atom by atom, from… nothing really. And this has worked for him in the past.

I give it about a 1 in 3 odds of success. It pretty much depends on everything being perfect and getting prying the logs out of our less-than-impressive host.
A 33% shot of seeing an egg unscramble is pretty good odds.

So yeah. Andrew Burton. He performs thermodynamic miracles.
(I’ll let you know how this one turns out.)

Most pirated TV shows

2009-02-25T00:37:55Z

The 10 most pirated Television shows on the internet February 9-15^* were;

Rank	Name	estimated downloads
1	Heroes	1,595,000
2	Lost	1,570,000
3	24	720,000
4	Desperate Housewives	555,000
5	Battlestar Galactica	515,000
6	The Big Bang Theory	500,000
7	Fringe	490,000
8	Grey’s Anatomy	455,000
9	Terminator: The Sarah Connor Chronicles	410,000
10	Dollhouse	375,000

^*based on BitTorrent file downloads, apparently

My response

I love someone is tracking this kind of data. (Even if it does omit direct downloads.) I wonder if it’s syndicated in an organized fashion?

Related note: Terminator is canceled. I am sad. (And a bit incredulous– why do you announce that before the movie hits?)

Sludge Test Economics

2009-02-24T04:51:23Z

Unrestrained Optimism - What could go wrong?

About three weeks into the 44^th presidency of the United States of America, not only is the Obama Administration a complete failure, but we’ve suddenly discovered that Hillary Clinton was the better candidate.

‹The sky will open. The lights will come down. Celestial choirs will be singing and everyone will know we should do the right thing and the world will be perfect!› —Hillary Clinton, Democratic primary race, 2008

Every time I think the American news cycle can’t get any stupider…
I sometimes feel like like America’s major news outlets are being run by that fish from Finding Nemo who can’t remember anything more than 2 minutes ago, so whatever thought that flitting through her mind at any given second is the most profound and important thing ever. Watching an hour of American news coverage leaves you dumber and less well-informed than you were when you started.

The one-track headline of the current news cycle is the failure of the Obama Administration to bring change to the country. And I confess that while I never much bought into the Hope & Change platform, my grin grows feral to see it counted out… because it’s just gaining steam. Every institution that’s insulated itself from economic feedback cycles through debt and bets against margin suddenly finds it can’t get credit through a hard time. Programs allowed to languish in ineffectuallity for decades now face do-or-die pressure. 30 years of accumulated fat; social, economic and institutional is set to burn off in an economy that, for the next two-to-three years, cannot afford to keep underwriting nonperformance.

Take the news. Public criticism of the state of the American newsmedia in the last decade has shifted from “it is biased” to “it is so incredibly bad,”and it’s gotten to the point that no one really disagrees with that latter statement anymore. American media outlets are mired in system-wide self-reinforcing behaviors so compelling that they short-circuit any corrective market tendency to reward better media— it’s all bad. CNN is just a better grade of clown.

Does anyone think the American newsmedia wouldn’t be improved by a few massive bankruptcies smashing the whole industry flat as a griddle cake and forcing them to concentrate on the neglected basics?
Or would we have a chance in hell of serious credit or health care reform without this meltdown? Because they’re gonna be the biggest issues in the 2010 election.

Maybe I’m just a ruthless optimist. But if the glass of sewage is half-empty ask yourself… did you really want the half that’s gone?

The Internet Safety Act

2009-02-21T23:55:56Z

Sentor John Cornyn is proposing a new federal law that would impose unprecedented data retention requirements on every business, individual and wi-fi hub in the United States of America. He wants your wi-fi modem to keep records of every time you signed on (even if it’s password protected) for two years– in order to “protect children.” (Key point: This kind of tracking would do almost nothing to track or prevent against child exploitatation– that kind of online interaction is real-time, but it’d be really handy for persecuting media pirates after-the-fact!)

Ever notice how Republicans are against any sort of “onerous” data retention where big businesses are concerned, but all for it when ti comes to the private individual? I have!

This kinda shit make me want to track down whoever put the bad magic in Vundo and sic them on US politics. Some situations just call for Weaponized Karma.

CNN uses Wordpress

2009-02-21T23:38:22Z

So there was a story on CNN.com that didn’t have an option to add comments, which irritated me. So I went to track down a story that would let me add comments, figuring I’d just change the story-ID using firebug and it’d probably accept my comment anyway. (You’d be surprised how often that works using monolithic content-management systems.)

And then my eyes crossed because I recognized this post-management structure. It turns out that the CNN comment system is powered by Wordpress, of all things! (It’s not a secret. There was even a ‘powered by Wordpress’ link at the very bottom of the page with the copyright notice and all the other crap no one reads.)

So, that’s no good for comment-hacking… the systems are isolated from one another. But Wordpress is a terrible system (I use it myself, I know of what I speak,) and I’m sure this fact will present opportunities for mischief in the future.

I’m just gonna file it in the back of my brain for now. And maybe sign up for a Wordpress security-alerts mailing list…

I hope not all dreams are metaphors

2009-02-21T20:27:36Z

I’ve been sick and I got up at 4AM and I was totally high and nothign was real so I thought I was dreaming but I wasn’t and that was weird because I hadn’t taken anything but I slept for fourteen hours before this and I haven’t eaten for two days so I thought it would go away but I just kept being high for liek five hours which was cool but also kinda scary and I watched the new Terminator episode which was really good before going back to sleep because I was getting the shakes.
I slept for about three hours and I had a dream about the Justice league being framed for stealing a statue by a polar bear posing as a security guard in the Happy Harbor museum. Only it turned out the polar bear was actually a grizzly bear in a polar bear costume. And the Grizzly bear was actually a pair of auto-cannibalistic Archaeopteryx (dinosaur-birds) who got arrested but like Greenpeace made us let them go because they were extinct and they were released back into the wild where they were killed by bear mobsters with chainsaws and Aquaman was the leader.

Anyway I’m mostly not-high now and that sucks.

On the structure of Record Albums

2009-02-16T18:44:23Z

My “why” phase was legendary. When I was in second grade, I synthesized Zeno’s Paradox and spent a very frustrating car ride trying to explain it to my mother. It was around the same time that I first noticed a nagging pattern in my parent’s old record albums; the “good” songs (an entirely subjective judgment) tended to be bunched together towards the beginning of the album, with maybe one decent song on the b-side. Graphed over time, the relationship would look like this:

I’m not a major audiophile, but this has always bugged me, because I could never figure out why the albums were patterned this way. …until last night.
The following is a transcript of a discussion between myself and Joe Rovang:

Me : *points finger in a vaguely revelatory manner*
Me : PHONOGRAPH RECORDS HAVE VARIABLE BITRATE!
Joe: Ah
Me : OR WHATEVER YOU CALL BITRATE WITH ANALOG SOURCES
Me : seriously, think about it!
Me : in geometry terms, on a CD, a 3 minute song might take, say, “14 inches” worth of length, wrapped around the disk, right?
Me : and that same 3 minute song gets 14 inches regardless whether it’s the first song or the last song.
Joe: Ah, I gotcha
Me : But on a PHONOGRAPH record though, the song at the beginning might have 24 inches, and the one at the end 8 inches! Because of the radius of the disk diminishes as the playnehead moves in, but the rotation speed does not accelerate to compensate!
Me : on CD’s it does. (or at least it does metaphorically– it’s not reading the data in realtime.)
Joe: Uh huh
Me : so that means that there is MORE audio information in the first track than the last
Me : CONSIDERABLY more.
Joe: Why, they’d have to vary the frequency whenever they made a new size of disc! Oh, but they did. :)
Me : the first track is higher fidelity, the last track is lowest.
Joe: Erm, I doubt that…
Joe: Well, actually, perhaps you’re right.
Me : I AM right
Me : records are rates in RPM
Me : rotations per minute
Me : that’s constant
Me : at the outer edge, the radius is larger. 22 RPM of that radius are liek 3 times as much data
Me : That BLOWS MY MIND
Joe: Well, I get that part, but I’m not sure the stretchiness affects the fidelity of the encoded audio.
Me : The tracks, if you read them out with a laser and played them at a rate corresponding circumference (? inches… of track?) to playback speed, the songs would get faster and faster as you played them
Me : “affect” in theoretical or practical terms?
Joe: Practical
Joe: :)
Me : in PRACTICAL terms, the “footprint” a particle of dust has on late-occurring tracks is mucht higher
Me : Pops become more severe, more noticeable, longer
Me : I grant you– analog medium. Any change in quality caused by the diminishing data density is probably below the level of human hearing…
Me : …if the player were in a vacuum box. But exposed to the environment, I argue, there actually IS a diminishment of quality.
Me : This makes so much SENSE
Me : I always wondered why albums don’t follow an act structure!
Me : I mean– you know, with a big finale
Me : instead the singles tend to be tracks 2-4… and the end song is NEVER a big finish except on concept albums where they’re deliberately imposing a structure
Joe: I’ve not found anything online to corroborate your theory.
Me : Tracks 2-4 are the SWEET SPOT, where music will sound BEST
Me : (track 1 would be abnormally susceptible to scratching, since you put the playhead down there.)
Joe: Hmm
Me : and– given that vinyl or plastic records are degraded every time you play them…
Joe: Well, it SOUNDS plausible
Me : putting the most-listened-to tracks in the high-density area maximizes their wear resistance!
Me : …that is so COOL!
Joe: Hmm
Joe: Well, I can’t find anything to disprove your reasoning, so I’ll go with it for now.
Me : I mean– I’m not a big music guy… but the structure albums seem to follow has always BUGGED me. I could never figure out why the “good songs” seemed to statistically cluster like that
Me : Not the first OR last, but not spread out evenly either.
Me : Musicians probably aren’t even aware they’re doing it anymore… it’s just TRADITION.
Me : Albums that had their good songs arranged in the sweet spots would sound slightly better, get a better response, and thus their structure influenced how subsequent musicians structured their albums!
Joe: Sure
Me : this tiny statistical pressure has, over time, created a self-reinforcing “rhythm” to how an album “should” be structured…
Me : that remains in effect even after we switched to CD’s where this isn’t an issue– an artifact of the originating medium!
Me : *grabs Joe by his shirt collar*
Me : DON’T YOU SEE HOW AWESOME THIS IS?
Me : THINGS MAKE SENSE!
Joe: Yes, that’s pretty cool
Me : …your lack of enthusiasm disappoints me
Me : haven’t you ever noticed patterns in the world around you that baffle you? You look at them and you’re just- “But that MAKES NO SENSE! Why is it that way? And it’s that way EVERYWHERE. It’s too consistent not to be real!”
Joe: Yes, your discovery is pretty cool.
Me : THERE IS A REASON FOR HOW ALBUMS ARE PACED!
Me : <– is happy
Joe: I approve, I do

This unexplained pattern has itched at the back of by my brain for twenty years.
It feels good to scratch.

Subprime at Fault — All Aboard the Partizanship!

2009-01-02T11:08:19Z

Ah, the holidays! A time to get together with families; chat, catch up and when the subject turned to politics-- argue.

My seasonal favorite of late has been "Clinton caused the Sub-Prime mortgage Crisis," referring to changes made to the Community Reinvestment Act in 1994 which opened up more sub-prime mortgages. My problem is I don't buy it. Mortgage success or failure do have a "delay" built into them-- you have more chances to fail over 30 years than you do over 3 years... but they don't have a 14 year delay. If a mortgage will fail it's more likely to do so early than late. People don't go into foreclosure on year 28 of a 30 year mortgage.

A typical example of this thinking: The Roots of the Subprime Mortgage Mess Have Clinton All Over Them goes by the logic that "Subprime Mortgages increased 10-fold after Clinton did his thing, therefore it's his fault." It's a compelling argument on its face-- until you realize that they were increasing from near-zero. Any increase would be dramatic. And those numbers dont' say anything about failure rates-- if the failure rates on those new mortages were so high... why isn't the axe-grinder trumpeting them?

After having had this argument for the umpteenth time with my father-- ending (again) with "you don't know that, you're just guessin," I set out to find proof. My thesis: Though the 1994 changes in the law did open the door to issueing more sub-prime mortgages, there were still checks and balances and oversight built into that system. For the current crisis to exist those mechanisms must have been undermined, and I think that occurred on the watch of our notoriously anti-oversight, anti-regulation 43rd President.

I want a dataset that goes back to 1994-- but I can't find one. Every single chart of the subprime mortgage market starts in the year 2001. ...that's funny, why would every statistician begin their chart with the year Clinton left office if the problem started on his watch? (Math must be in the tank for Clinton!) Despite this, I was able to find some useful visualizations href="http://faculty.baruch.cuny.edu/lwu/seminar/vanHemert_slide.pdf">here</a>.

Subprime Mortgage foreclosures by year. Please note-- X is "months" not time, so these lines should actually be strung out horizontally. We had bumper bad years in 2001 and 2006.

2001 I can get- after 9/11 lenders were eager to give anyone a loan. And in the urge to kickstart the economy requirements were loosened. But why 2006? This is a bit interesting...

Gross Subprime mortgage data by year (highlighting mine)

Wow, check those volumes, and amounts! And in 2006, the "hell year" where the number of failures doubles... balloon payments suddenly came back huge! The "you don't have to pay for awhile but then you have to pay a lot" financing structure came into vogue-- and people started losing their houses on an epic scale.

So that's starting to look like an indicator- and the "balloons-to-0" trend of 2004 kinda looks like reckless loan terms by mortgage brokers that didn't expect to be servicing their own loans... but still, the dataset is incomplete. I need the year 2000 and earlier!

Well, this is the best I could find. (Blatantly filched off Wikipedia.)

Mortgage loan fraud by year

Now fraud isn't a perfect indicator-- but for the purposes of this argument I'm willing to say "you only catch fraud if there's a foreclosure." And since what I'm really searching for in these numbers is signs that the oversight and regulations that used to function were de-fanged 2001-on, fraud is a decent metric.
Oh dear. Look at those numbers! And a sharp spike in 2005-- hey, I bet that's what caused the return to balloon payments in 2006! A number crop of outright fraud causes pressure to secure these loans somehow-- a tangible sort of anchor speaking to their financial "reality," like a large lump-sum payment due in two years!

*sigh* Look, the "Clinton did it" argument is stupid. We lived through 2001-2006. We remember when suddenly banking and mortgage commercials were everywhere. That is when the industry underwent hothouse growth.
Or as the NYU school of business paper I liked to before states in its abstract; "Quality loans deteriorated year-after-year since 2001"

Clinton "Opened the door" in 1994... but for 6 years there was no such deterioration. Oversight was doing its job. He left office, and the foxes were guarding the hen-house, with predictable results.

Clearly I don't buy the Clinton-causal thesis. Or the idea that the economy works on an 8 to 16 year lag, which my father also like to trumpet because it means all Clinton's success was Regan and Bush's doing, and W's failings are Clinton's fault-- another stupid idea that gets hammered along partisan lines.
There was a terror attack in 2001 and a lot of regulations were gutted to try and jump-start the American economy! That doesn't make the subsequent mortgage meltdown Bush's fault... but it both happened and originated on his watch- trying to make it a partisan thing by saying Clinton did it is both incorrect and misses the larger issue.
Yeah, Republicans are anti-regulation... (I even buy into this trap,) but removing them kinda made sense, at the time. Democrats tend to look down on Economic Stimulus Packages-- but it looks like Obama's going to be passing one of the biggest ones ever.

You can fault Obama is he fails to ride the consequences of his package closely enough-- but for either of them-- enacting the changes they have to at the time isn't-- and cannot be-- a partisan issue.

Stop making it one.

Also your argument doesn't hold up, go away.

Worrisome new Livejournal behavior...

2008-12-31T12:22:26Z

For the lat month or so, I've been noticing something worrying. I'll click on a link-- and be taken to the Livejournal login screen.
I wasn't sure at first... maybe I was clicking on something else by accident? (The place I ran into it most often was very 'busy' with many links in a small area.) But today it happened somewhere else-- where there was only one clickable link and nothing else I could have hit by accident. I clicked on a link to someone's blog and was instead taken to the livejournal login screen. Livejournal wanted me to log in before it would let me read a publicity-accessible blog.

I suspect this is Livejournal's idea of being NICE. After all, I was logged into Livejournal-- I mist want to type in my information and be logged in again, right?
(No, because I use the Livejournal login plugin for Firefox.)

On a technical level- I suspect this involves cookies or sessions rather than IP addresses. Upon requesting someone's blog, if Livejournal notices- "Hey, last time we saw this guy he was logged in, and he never manually requested we log him out" instead of serving me the page I actually requested it sends a 307 header to my browser and redirects me to their login page.

Here's the thing... I surf in tabs. Especially while searching for something I might open dozens of tabs, then go through them in order until I find what I want. If I switch to the next tab and see a livejournal login window-- I'm going to close that tab. There's nothing about these login windows that says "Ha ha, we were just fooling you, you don't have to login... but if you did we'd give you a different page you were actually looking for!" so as far as I'm concerned-- this is junk. I will never see that page because Livejournal has jogged my elbow.

And while I appreciate that they're trying to be nice for some reason? It's a webpage. It has an explicit address. If I request that page I want to be shown that page, not asked to log in first.
Logging in is completely optional by the way, if I click "back" then hit the link again I'm taken to the page I requested even if I'm still not logged in.

I think I understand why Livejournal is doing this (completely aside their greater ability to track and thus monetize the activities of logged in users,) but they shouldn't.
They are embracing a system where you request a page and are magically taken to a different page... sometimes. That level of arbitrariness is very, very bad. That is not how the internet should work. A 307 code means "page has temporarily moved." Browsers allow 307 codes despite the vast potential they have for abuses like this because they are mostly used for good, and used in ways that users never notice.

Javascript isn't allowed to open new windows anymore. It's not a perfect parallel... I'm just sayin'.

Anyway I suspect that if I was never a Livejournal user I wouldn't be exposed to this stupid elbow-jogging... but having experienced it as a background irritant for weeks before actually "catching it in the act," I am reminded of Wikia's unfortunate penchant for spamming anonymous users with demands that they log into their site just to view it.

I'm sure LJ's intention is quite different... but the mechanism they have chosen to enact that intention is even more annoying than Wikia's, and I am disinclined to read it generously.

Update: I have now stayed logged out of Livejournal for a week, and I can definitively say that Livejournal is much less irritating for unregistered losers. Not once was I 307'd to a login page while logged out. But when I logged in to post on a friend's journal, it was just a few hours later that I ended up on their damned login screen again for asking to view (not post, view!) someone else's journal!

The trigger of this obnoxious behavior hinged on the type of login used.
1) If you were logged in but had "remember me" checked, the system doesn't give you shit (because you never got logged out-- "Remember Me" remembers you indefinitely.)
2) But- if you did not choose to have Livejournal remember you, once your session expires (by closing the browser window for example,) you will no longer be logged in. But, the next time you go to the Livejournal site, it reads the cookie from your browser and says "Hey, I remember you! You're the guy who didn't want me to remember him! *bounce* Yeah, I know you requested another web page, but I'm gonna send you to this one instead and demand you log in before I'll show it to you! I will not provide a link to skip past this... the only way to actually get to the link you actually requested is to click back in history, request the link again and hope Livejournal lets you through this time!"
3) If you fully logged out (or were never a member,) the system does not give you shit.

Caveat:
4) You can also "bind cookie to IP address" (A sort of "remember me" that logs you out when your computer changes location.) I'm not cure how this would affect things.

Clearly this behavior irritates me. but is it actually a big deal? I mean- for all the negative slant I put on it... what #2 really means is that Livejournal is prompting you to re-enter your login information if you were passively logged out by closing your browser. LJ's operating logic seems to be- "If the user didn't actually click the 'Log Out' links, they probably want to be logged in. Let's ask them to log in again!"

This is the fundamental problem I have with that;
IF I DIDN'T WANT MY LOGIN SESSION TO END WHEN I CLOSED THE BROWSER, I WOULD HAVE SELECTED "REMEMBER ME."

I DO NOT WANT TO BE LOGGED IN FOREVER.

LIVEJOURNAL IS ASSUEMING THAT USERS WHO OPTED NOT TO SELECT "REMEMBER ME" ACTUALLY DO WANT TO BE REMEMBERED, AND STAY LOGGED IN AT ALL TIMES.

IF I WANTED TO BE LOGGED IN FOREVER, I WOULD HAVE SELECTED "REMEMBER ME."

I DID NOT. STOP TREATING ME LIKE I DID.

*ahem* So... yeah. This new functionality is pretty wrongheaded at inception. And the actual mechanism for implementing it gives me the screaming meemies. This is a typical example of the people inside the project having a skewed perspective on their own work. 4Apart employees are always (or at least presumably always want to be) logged in to livejournal. They would encounter this behavior only rarely and it would not be irritating.
I have no desire to be logged in 100% of the time. I encounter this prompt daily. And it's maddening to be rapidly clicking through something-- and be prompted to log in before viewing something you do not have to be logged in for.

For the heavy Livejournal user, I can see the appeal of this "feature." It prevents you from accidentally positing comments while logged out. And being forced to log in every time your session restarts is a major security/privacy bonus vs. always being logged in-- especially if you share a computer!

I get that, I understand that. But I don't want that, and my use of this sits has suddenly become 800% more annoying to the point that I find it preferable to stay logged out all the time and post anonymously rather than deal with this shit. (I know it's a little thing, but really... it's like stroking a cat's fur backwards, being slapped in the face with a login page seriously jolts me out of the zone.)

I have no problem with Livejournal providing that functionality for the people who want it- even making that the default behavior... but provide me a way to turn it off. Because (and I know this is getting old) if i wanted to be logged in all the time, I would have checked "Remember Me." I did not, so stop remembering me and preventing me from using your site.

Mirroing a remote directory with PHP

2008-12-29T17:53:55Z

PHP time, yay!

This is a project I created to filfill a specific need; I wanted to mirror the contents of a directory. This was a publicly-accessable, listable directory on a corporate web site. It contained sub-directories, which in turn contained product images. My script anticipates that very rigid file structure-- if your directory is different you'll need to make /major/ adjustments to your code.

I'm saving this as "mirror.php" First the backbone;

<?php
$_HOME = "./";
include_once('includes/buckler.php');

echo '<p><a name="end">END ANCHOR</a></p>';

My "buckler.php" file is a general include I use on almost every script I write. It sets a bunch of blobal parameters, loads up a templating system, some classes i use frequently etc. I could go in and turn all those off... but it's a lossy one-time use. I'm not concerned about efficiency.
The only buckler-loaded resource I'm going to use is the Snoopy php class. This is a class that simulates a web browser. Using the Snoopy class gives me flexability in case I run into problems-- like a server that checks for cookies, referrer etc... Snoopy can forge all of those with ease. (For this project all the requests turned out to be passive requests that could have been done with file_get_contents(). Oh well!)
Without the buckler, the first 2 lines would look like this;
include_once('Snoopy.php');

I'm running this script on a very cheap shared web-host rather than my own server, and they kill my scripts if they go too long so I'm going to do this in stages. I'm going to set up some "controls" for the script, as well as some global variables.;

echo "<p><a href=\"./mirror.php\">[home]</a></p>\n\n";

$MIRROR_DIR = "mirror/";
$FROM_DIR = 'http://www.buynlarge.com/common/portals/productimages/';

switch ($_REQUEST['mode']){
	case 'root':
		break;
	case 'sub_dir':
		break;
	case 'delete_dirs':
		break;
	default:
		echo "<pre>\n\n";
		echo '<a href="?mode=root">Mirror root dirs</a>' . "\n\n";
		echo '<a href="?mode=delete_dirs">Delete root dirs</a>' . "\n\n";
		echo '<a href="?mode=sub_dir#end">Mirror subDirs</a>' . "\n\n";
}

I've created a [home] link that always appears on the top of the page. $FROM_DIR is the original product directory, and $MIRROR_DIR is my logcal directory I'm copying things to. A switch statement forms the backbone of my script-- all the different modes of the document are controlled by the Request string. So "mirror.php?mode=root" will (eventually) cause it create all the directories I want. The "default:" case creates a set of controls that only show up when I'm at [home].
The "Mirror subDirs" link takes you to the "#end" anchor at the bottom of the page. We're going to have some long lists printing out, this takes us right to the summary.

Let's look at the 'delete_dirs' case. It's very short, and it deletes all the directories in our local mirrored root. (Well, assuming they're empty, otherwise it throws an error.) I used this class during debugging.

case 'delete_dirs':
		foreach (glob('./' . $MIRROR_DIR . "*") as $filename) {
			@rmdir($filename);
		}
		break;

VERY short. This uses glob to get an array of all files and directories matting a standard DOS pattern-- in this case "./mirror/*". You could get all .txt files with "./mirror/*.txt".
Since I KNOW everything here is a directory I dont' bother with is_dir or is_file, I just do a foreach and remove EVERYTHING. The "@" before rmdir suppresses error if you try to delete a directory with somethign in it-- you'll have to clean those up by hand.

Our code now looks liek this;

<?php
$_HOME = "./";
include_once('includes/buckler.php');

echo "<p><a href=\"./mirror.php\">[home]</a></p>\n\n";

$MIRROR_DIR = "mirror/";
$FROM_DIR = 'http://www.buynlarge.com/common/portals/productimages/';


switch ($_REQUEST['mode']){
	case 'root':
		break;
	case 'sub_dir':
		break;
	case 'delete_dirs':
		foreach (glob('./' . $MIRROR_DIR . "*") as $filename) {
			@rmdir($filename);
		}
		break;
	default:
		echo "<pre>\n\n";
		echo '<a href="?mode=root">Mirror root dirs</a>' . "\n\n";
		echo '<a href="?mode=delete_dirs">Delete root dirs</a>' . "\n\n";
		echo '<a href="?mode=sub_dir#end">Mirror subDirs</a>' . "\n\n";

}
	echo '<p><a name="end">END ANCHOR</a></p>';

?>

This is the big, ugly one; mirroring the root directory. "case 'root':"
First we'll fetch the directory listing and save it to a variable $page. I'm using Snoopy here, you can use file_get_contents if you prefer.

		$snoopy = new Snoopy;
		$snoopy->fetch($FROM_DIR);

		$page = $snoopy->results;

Now we need to get all those directory links. I'm using preg_match_all() and Regular Expressions. Regular expressions can be very intimidating for first ime users, but they are well forth the time to learn.

		$matches = array();
		$pattern = '/<a href="(.+)">/Ui';
		preg_match_all($pattern, $page, $matches);

The pattern /<a href="(.+)">/ matches all the anchor tags, and makes a matches the url's they link to. U make this "ungreedy" (grab as little text as possible) and i makes it case insensitive. (i is probably unnecessary, but I used it anyway.)

The content matching the parentheses will go into an array located at $matches[1].

		foreach ($matches[1] as $dir ){
			$directory_uri = './' . $MIRROR_DIR . $dir;
			if (( !is_dir($directory_uri) ) && ($dir != '/common/portals/') ){
				mkdir( $directory_uri );
				echo '<b>DOES NTO EXIST:</b> ' . $directory_uri . '<br/>';
			} else {
				echo '<b>exists:</b> ' . $directory_uri . '<br/>';
			}
		}

If the matched address isn't already a directory and isn't '/common/portals/' (a root-relative link at the top of the page I found easier to just test for rather than remove) then I create it using mkdir! Finally I added some feedback. If you run this script more than once you get a list of all the directories tellign you which are/aren't created.

Since we've now mirrored the top level of the hierarchy, how do we do the second level? For "case 'sub_dir':" I'm going to iterate through every directory we just created and check if there's an arbitrarily named file 'count.txt' inside-- which indicates I've already dealt with this folder. If there isn't I'm going to "deal with it."
There are a LOT of folders here, so I only want to deal with one per "turn." So at the top of my document I'm adding $MAX_DIR_COUNT and a few other global variables.

$TOTAL_DIRS = 0;
$LAST_DIR_COUNTED = 0;
$MAX_DIR_COUNT = 1;

And at the very bottom I'm going to add a readout to tell me how many directories have been counted.
echo 'Last dir counted: ' . $LAST_DIR_COUNTED . ' of ' . $TOTAL_DIRS . ' ||';

This readout will show up as 0::0 on the other modes... but oh well!

		foreach (glob('./' . $MIRROR_DIR . "*") as $filename) {
			echo "filename: " . $filename . " ";
			if ( is_file($filename . '/' . 'count.txt') ){
				$LAST_DIR_COUNTED++;
				echo " COUNTED<br/>";
			} else {
				echo " <b>NOT COUNTED</b><br/>";
				count_dir($filename);
			}
			$TOTAL_DIRS++;
		}

Glob again, we're iterating along each directory within the root, incrementing $TOTAL_DIRS as we go.
We test "is_file" to see if there is a file named "count.txt," if there is we incremember $LAST_DIR_COUNTED (we've lready been here,) if there ISN'T we call the count_dir() function for this directory! (This function doesn't exist yet!)

One extra wrinkle, at the end of "case 'sub_dir':" I'm adding the following code.

echo '<head><meta http-equiv="refresh" content="5"></head>';

That's a meta refresh tag. (Strictly speaking it should go at the top of a document for HTML standards, but this was ugly code anyway.) This tag will cause the document to reload itself 5 seconds after it's done loading. We're going to use this mechanism to make go through all our directories-- you just open the script in a window and go away for awhile, eventually it'll load everything.

This is what mirror.php looks like now:

<?php
$_HOME = "./";
include_once('includes/buckler.php');

echo "<p><a href=\"./mirror.php\">[home]</a></p>\n\n";



$MIRROR_DIR = "mirror/";
$FROM_DIR = 'http://www.buynlarge.com/common/portals/productimages/';

$TOTAL_DIRS = 0;
$LAST_DIR_COUNTED = 0;
$MAX_DIR_COUNT = 1;


switch ($_REQUEST['mode']){
	case 'root':
		$snoopy = new Snoopy;
		$snoopy->fetch($FROM_DIR);

		echo "<pre>";

		$page = $snoopy->results;

		//print_r( $snoopy );

		$matches = array();
		$pattern = '/<a href="(.+)">/Ui';
		preg_match_all($pattern, $page, $matches);

		//print_r( $matches[1] );

		foreach ($matches[1] as $dir ){
			$directory_uri = './' . $MIRROR_DIR . $dir;

			if (( !is_dir($directory_uri) ) && ($dir != '/common/portals/') ){
				mkdir( $directory_uri );
				echo '<b>DOES NTO EXIST:</b> ' . $directory_uri . '<br/>';
			} else {
				echo '<b>exists:</b> ' . $directory_uri . '<br/>';
			}
		}

		break;
	case 'sub_dir':
		foreach (glob('./' . $MIRROR_DIR . "*") as $filename) {
			echo "filename: " . $filename . " ";
			if ( is_file($filename . '/' . 'count.txt') ){
				$LAST_DIR_COUNTED++;
				echo " COUNTED<br/>";
			} else {
				echo " <b>NOT COUNTED</b><br/>";
				count_dir($filename);
			}
			$TOTAL_DIRS++;
		}

		echo '<head><meta http-equiv="refresh" content="5"></head>';

		break;
	case 'delete_dirs':
		foreach (glob('./' . $MIRROR_DIR . "*") as $filename) {
			@rmdir($filename);
		}
		break;
	default:
		echo "<pre>\n\n";
		echo '<a href="?mode=root">Mirror root dirs</a>' . "\n\n";
		echo '<a href="?mode=delete_dirs">Delete root dirs</a>' . "\n\n";
		echo '<a href="?mode=sub_dir#end">Mirror subDirs</a>' . "\n\n";

}


	echo '<p><a name="end">END ANCHOR</a></p>';
	echo '<p>Last dir counted: ' . $LAST_DIR_COUNTED . ' of ' . $TOTAL_DIRS . ' ||</b>';

?>

Finally we have a series of simple functions, starting with count_dir(). These can live at the bottom of mirror.php.

count_dir() was called because the directory we're passign it didn't have a "count.txt" file in it. So it's going to request the cousin-directory off the remote server and get a list of all those files for us! Lets get started, first some globals.

	function count_dir($filename){
		global $MAX_DIR_COUNT, $FROM_DIR;
		if ($MAX_DIR_COUNT > 0){
			$dir_name = null;

			/* Grab the directory name */
			if ($dir_name){
				/* Count directories */
				$MAX_DIR_COUNT--;
			}
		}
	}

$MAX_DIR_COUNT was our global that controls how many directories we're gonna attempt to count per reload (1). This function is called for every uncounted directory, but it doesn't do anything if the number of counts "allowed" is less than 1. (More inefficient construction in our 1-off single-use code, yay!) If it does do something it decrements $MAX_DIR_COUNT (down to 0) so that it won't do anything NEXT time it's called.
First we'll be getting the name of the directory (it's a bit messy at the moment) "$filename" is actually a full local directory. "./mirror/012345-2A" or somesuch. You could explode this and grab the last item, but I'm using pregs again.

			$matches = array();
			$pattern = '/[0-9A-Z-_]+$/Ui';
			preg_match($pattern, $filename, $matches);

The brackets mark a character set-- in this case any character 0-9, or A to Z, or a dash or underscore. The "$" signifies the end of the string being searched-- so this pattern will match the longest alphanumeric (plus dash & underscore) string it can at the end of the string-- which is to say after the "/".
...explode() probably would be easier. Oh well.

if (isset($matches[0])) $dir_name = $matches[0];

This is a completely unnecessary (read: necessary) bit of safety code-- if it can't find anything that looks like a directory name, $dir_name will remain null and the script will shrug and move on without accidentally tryignt o write to our root directory or anything. I don't think there's any circumstance where this would happen, but better safe than sorry.

$dir_name in hand, we now fetch the listing for that directory off the remote server. Snoopy again.
$snoopy = new Snoopy; $snoopy->fetch($FROM_DIR . $dir_name . "/"); $page = $snoopy->results; echo '<pre>';

I echo'd a <pre> tag because we're going ot be printing some debug code shortly. Next is Regular Expressions, again. Oy!

$matches = array(); $pattern = '/<a href="(.+\..+)">/Ui'; preg_match_all($pattern, $page, $matches);
This variant specifically searches for links that have a "." in them that is NOT the first character. (It have to be escaped with a "\") This excludes the "up a level" link-- that's not a file! (I think. As I write this that doesn't seem to quyite make sense, but the code works so I'm not going to worry about it.) Likewise any sub-directories (lacking .'s) would not match. We're looking for files with names like "123.jpg"
print_r($matches[1]); echo '</pre>'; count_files($matches[1], $dir_name);

We do a print_r on the results, then toss them (along witht he directory name) to the non-existant function count_files()!
That's it for count_dir(). The full function looks like this;
function count_dir($filename){ global $MAX_DIR_COUNT, $FROM_DIR; if ($MAX_DIR_COUNT > 0){ $dir_name = null; /* Grab the directory name */ $matches = array(); $pattern = '/[0-9A-Z-_]+$/Ui'; preg_match($pattern, $filename, $matches); if (isset($matches[0])) $dir_name = $matches[0]; if ($dir_name){ $snoopy = new Snoopy; $snoopy->fetch($FROM_DIR . $dir_name . "/"); $page = $snoopy->results; echo '<pre>'; $matches = array(); $pattern = '/<a href="(.+\..+)">/Ui'; preg_match_all($pattern, $page, $matches); print_r($matches[1]); echo '</pre>'; count_files($matches[1], $dir_name); $MAX_DIR_COUNT--; } } }

count_files() is slightly more complicated than it has to be-- rather than simply creating the file "count.text" it's gonna put the name of all the files inside it, each on a seperate line.
I don't actually use this functionality anywhere, but I also don't have more than 20 files in any of the directories I'm copying. If you were copying 150 files you'd probably have hit the directory in multiple interations. Then this file would be useful. (Probably.)

function count_files($file_array, $dir_name){ global $MIRROR_DIR, $FROM_DIR; $count = count($file_array); echo "I counted $count files! "; $content = ''; foreach($file_array as $file){ $content .= $dir_name . '/' . $file . "\n"; } get_files($file_array, $dir_name); file_put_contents ( './' . $MIRROR_DIR . $dir_name . '/' . 'count.txt', $content); }
So it's a really straightforward file_put_contents() command. Note that I call get_files() (our last function) before I create the "count.txt" file. This is so that if anything goes wrong when I'm actually copying the files with get_files() and the script errors out, the count.txt file won't have been created yet, and the script will take another crack at it.

Last function, woohoo!

function get_files($file_array, $dir_name){ global $MIRROR_DIR, $FROM_DIR; foreach($file_array as $file){ $fetchedFile = file_get_contents($FROM_DIR . $dir_name . '/' . $file); file_put_contents ( './' . $MIRROR_DIR . $dir_name . '/' . $file, $fetchedFile); } echo " " . count($file_array) . " files fetched successafully."; }
Yes, after all that drama, that is exactly as inefficient as it looks. file_get_contents() is a horrible, horrible, wonderful slacker shortcut-- it loades the entire file into memory. If the file is too big it'll crash or lock up the script server. These are all smallish .jpg files (150k) so, again... good enough. If you were gloming .pdf files or anything over a megabyte you'd need to use fopen or something.

That's it! Click "Mirror root dirs" to copy over the top-level file structure. When you click the "Mirror subDirs" link it's going to start going over every directory one at a time, snag the files from the remote server and save them locally. 5 seconds after it's done, it'll reload itself and do it again for the next directory. Come back in a couple hours and it'll be done-- still reloading itself but showing all the directories are all accounted for.

In my case I'm mirroring 3500 product directories with an average of 15 images inside each directory. As of this writing my script has been running for ~2 hours and is about 10% done.

Total time to write the script, including distractions, ~1 hour.
I've been doing too much Actionscript... the ability to just do something messy and quickly is so freeing! Nothing takes less than 4 hours in Actionscript, and you have to pre-plan the architecture of everything... I love PHP.

Full script of "mirror.php" is as follows;
<?php $_HOME = "./"; include_once('includes/buckler.php'); echo "<a href=\"./mirror.php\">[home]</a>\n\n"; $MIRROR_DIR = "mirror/"; $FROM_DIR = 'http://www.buynlarge.com/common/portals/productimages/'; $TOTAL_DIRS = 0; $LAST_DIR_COUNTED = 0; $MAX_DIR_COUNT = 1; switch ($_REQUEST['mode']){ case 'root': $snoopy = new Snoopy; $snoopy->fetch($FROM_DIR); echo "<pre>"; $page = $snoopy->results; //print_r( $snoopy ); $matches = array(); $pattern = '/<a href="(.+)">/Ui'; preg_match_all($pattern, $page, $matches); //print_r( $matches[1] ); foreach ($matches[1] as $dir ){ $directory_uri = './' . $MIRROR_DIR . $dir; if (( !is_dir($directory_uri) ) && ($dir != '/common/portals/') ){ mkdir( $directory_uri ); echo 'DOES NTO EXIST: ' . $directory_uri . ' '; } else { echo 'exists: ' . $directory_uri . ' '; } } break; case 'sub_dir': foreach (glob('./' . $MIRROR_DIR . "*") as $filename) { echo "filename: " . $filename . " "; if ( is_file($filename . '/' . 'count.txt') ){ $LAST_DIR_COUNTED++; echo " COUNTED "; } else { echo " NOT COUNTED "; count_dir($filename); } $TOTAL_DIRS++; } echo '<head><meta http-equiv="refresh" content="5"></head>'; break; case 'delete_dirs': foreach (glob('./' . $MIRROR_DIR . "*") as $filename) { @rmdir($filename); } break; default: echo "<pre>\n\n"; echo '<a href="?mode=root">Mirror root dirs</a>' . "\n\n"; echo '<a href="?mode=delete_dirs">Delete root dirs</a>' . "\n\n"; echo '<a href="?mode=sub_dir#end">Mirror subDirs</a>' . "\n\n"; } function count_dir($filename){ global $MAX_DIR_COUNT, $FROM_DIR; if ($MAX_DIR_COUNT > 0){ $dir_name = null; /* Grab the directory name */ $matches = array(); $pattern = '/[0-9A-Z-_]+$/Ui'; preg_match($pattern, $filename, $matches); if (isset($matches[0])) $dir_name = $matches[0]; if ($dir_name){ $snoopy = new Snoopy; $snoopy->fetch($FROM_DIR . $dir_name . "/"); $page = $snoopy->results; echo '<pre>'; $matches = array(); $pattern = '/<a href="(.+\..+)">/Ui'; preg_match_all($pattern, $page, $matches); print_r($matches[1]); echo '</pre>'; count_files($matches[1], $dir_name); $MAX_DIR_COUNT--; } } } function count_files($file_array, $dir_name){ global $MIRROR_DIR, $FROM_DIR; $count = count($file_array); echo "I counted $count files! "; $content = ''; foreach($file_array as $file){ $content .= $dir_name . '/' . $file . "\n"; } get_files($file_array, $dir_name); file_put_contents ( './' . $MIRROR_DIR . $dir_name . '/' . 'count.txt', $content); } function get_files($file_array, $dir_name){ global $MIRROR_DIR, $FROM_DIR; foreach($file_array as $file){ $fetchedFile = file_get_contents($FROM_DIR . $dir_name . '/' . $file); file_put_contents ( './' . $MIRROR_DIR . $dir_name . '/' . $file, $fetchedFile); } echo " " . count($file_array) . " files fetched successafully."; } echo '<a name="end">END ANCHOR</a>'; echo 'Last dir counted: ' . $LAST_DIR_COUNTED . ' of ' . $TOTAL_DIRS . ' ||'; ?>

Committing identity theft for teaching purposes.

2008-12-07T03:32:02Z

My "Futures" class has a required project dealing with "a future issue." I chose Identity Theft.

I have very little interest in Identity Theft as it's usually addressed in the media-- I think it's reported on in a supremely unhelpful manner. Instead, I decided to redress this weakness by performing a case study. Specifically, I stole the records of one million people, and demonstrated how easy it was to get into their bank accounts.

Over a year ago, I ran into an online game (which shall remain nameless) whose userdata was very, very poorly secured. Epically so. I even spoke to the developers about it; they acknowledged the problem, said they intended to fix it... but it never got fixed. A year and a half later- a million people have signed up to play it.

Thing is, 32% of those people used the SAME password for the game that they used for their e-mail account. So if someone (me) were to hammer the game's server... he'd end up with access to 300,000 people's e-mail account.

It made for a really impressive display, I printed out all one million names and used them to wallpaper a 16' x 10' hallway, which I then covered with notes about my methodology and conclusions.

I used David Willis as an example-case when presenting the project to my class. Here's a guy who makes his enetire living online- but he used the same password for his e-mail account that he did for the game! (A snapshot of David Willis' Hotmail inbox from that morning made this "real" for my classmates.) From there it's just one "your statement is ready" into his bank, credit card or paypal account-- even if he bothered to use a different password for them "I forgot my password" sends messages back to the compromised Hotmail account!

I'm gratified to report that my entire class synthesized the lesson I was getting at before I even had to say it. We all worry about our bank passwords... which we precieve as needing "higher security." Very few stop to realize that your primary e-mail account (usually secured by a 'lower priority' password you use on many things) is a pathway 'up into' those accounts.

When your bank, or social security or whatever... have a security breech, they will tell you, and be on the watch for suspicious activity. Little piddling online game sites won't. And they don't feel like they have to properly secure their data-- after all it's not like there's anything important about your game score, right? But because people re-use passwords... that unsecured data can be used to compromise other systems which are sensitive.

There's no lessons here that couldn't be learned through socratic discussion-- but having a tangible example to look at made the whole scenario obvious to everyone I was presenting to... their paradigm got shifted to a man on the first try, and what had previously been a sort of abstract-fuzzy concept of identity theft... suddenly made total sense to them, on almost every level. Their conclusion: Have a password for your primary e-mail you don't use anywhere else, it is what you really need to worry about securing, more than your bank account because your e-mail provides access to your bank account!

I mentioned to David that I'd used him as an example in a presentation, which he seemed to find amusing, and I later posted a link to pictures... but I don't think he ever looked to see what kind of example I meant.

Oh well, I'm sure the news will get back around to him. Somehow.

(I should probably be giving David a hard time for making this kind of mistake... he lives and dies by the net, but I made the same mistake. We both have our most-commonly-used passwords sitting out in this incredibly insecure system, waiting for someone less pure-hearted then myself to come along and mine the data out... and the game in question provides no mechanism for changing that password. *sigh*)

Update: 10 days after this blog entry went online, the game in question was abruptly shut down. The reason for the closure is unknown, it had received a code update just 30 days before.

deriksmith @ 2008-11-04T11:30:00

2008-11-04T17:32:04Z

A love song to Archive.org

2008-11-03T23:08:24Z

Archive.org, how do I love thee?

Today, while developing code for my first personal web site since the year 2000, I used Archive.org to yoink some Javascript code from el Fuñaroverse, circa 2004.

What cracks me up? Back when el Fuñaroverse was launching, I used Archive.org to yoink this same code from Derik's PR-Themed Site of the Week, my aforementioned defunct dating-from-1998 personal website.

I am now using Javascript that I wrote more than 10 years ago with almost no changes. I was terribly proud of it at the time, and I'm even prouder to say that it still stands up a decade later. No re-coding needed.

Thank you Archive.org, you made me more awesome.