Hitskin.com
Home
Thanks to a community "white hat," the RIFT account
security exploit (that had nothing to do with ZAM) has been squashed.
Read our exclusive interview here!
For the past two weeks, the RIFT community
has been rocked by a seemingly endless onslaught of accounts being
compromised. Account security is an incredibly touchy subject with MMO
players, and once it became apparent that this was an epidemic rather
than a handful of occurrences, fansites like ZAM were blamed as part
of, as Trion put it, a “witch hunt” by some members of the community.
It turns out that fansite security had nothing to do with the whole mess.
The true culprit behind the account debacle
was tracked down by a member of the RIFT community that ignored all of
the speculation going on around him and put real effort into making
RIFT a safer place to play. Known only as ManWitDaPlan on the RIFT forums (and Webmaka on ZAM’s own forums), he exposed the flaw with the aid of several other committed RIFTers. Trion quickly found his post, contacted him directly, and within hours had the exploit fixed. The entire community, including all of us at ZAM, breathed a huge sigh of relief.
UPDATE: Trion speaks out and clears the air about ManWitDaPlan's find.
Over the last few hours, we’ve managed to reach out to MaWitDaPlan and
get his side of the story, find out what his background is, and see
what he thinks of the future of RIFT. We hope you enjoy this exclusive
interview!
ZAM: What’s your background? Do you often work with account security?
ManWitDaPlan: I've
been a programmer for most of my life (started at age ten), and
professionally so since 1995. I currently own a small security-software
company specializing in secure data destruction. I'm also something of a
"white hat" (aka "ethical hacker") in that I was involved with the
cracking and warez scenes in my younger days and have basically employed
the skillsets from that era of my life for more worthy pursuits.
ZAM: Why did you want to pin it down? Did you get an account hacked? Or did you just feel like doing a good deed?
ManWitDaPlan: My
account was hit during the start of the hack-fest on the weekend of the
12th-13th of March. I was left with only two pieces of armor and some
gold, but my bank and mailbox were completely untouched, which deviates
from the norm for what happens to hacked accounts. That made me
immediately suspect there was a bug of some sort.
Since I've been
working with security for so long I run a very, VERY locked-down system
and knew that there was pretty much no way I could have been hacked via
the usual avenues (e.g., malware). Still, I invoked some of my more
aggressive anti-malware scanners, including a couple that act as
hypervisors, and every scan I could throw at the systems turned up
nothing at present, and no signs of ever having been infected in the
past.
Between these two - the bank being untouched and the
systems being clean - I started looking up the chain from me to the game
servers proper. I started searching for man-in-the-middle attacks and
server-side compromises. That avenue began with investigating how the
game works, which led to finding an exploit, which led to working out
how it works, which led to the forum post that from the looks of things
shook all of Telara in a way that'd make Regulos go "umm, okay, let's go
find another planet to eat - these people are nuts!"
ZAM: There was a lot of finger pointing going on around the
community, especially towards ZAM RIFT and RIFT Junkies. What would you
tell those folks that were on, as Trion called it, a “witch hunt?”
ManWitDaPlan: Unfortunately
that sort of thing is normal human nature. 99.999% of the time a game
account gets hacked it was because the user of that account did
something dumb, like using weak passwords or reusing compromised
credentials, or just as frequently, allowing malware to get a foothold
and leech previously-safe credentials.
This time around, it was
an active, in-the-wild exploit, so the normal causes were not the
primary ones. Some people cannot remove the blinders of their own
preconceptions, though, and couldn't adjust to the idea that it wasn't
malware or poor client security in all cases, so they stuck with what
would be the most likely answer under normal circumstances.
Once it became increasingly clear there was more going on that just
bad/reused passwords and those folks were forced to rethink their "it's
got to be your fault" stance, the next target was anyone that ran
anything that could conceivably be a hacking vector. Thus, the finger
pointing toward fansites that offer any form of active content or
add-on. ZAM took an extra-tough dose of baseless blame thanks to the
old, no-longer-valid link to RMT companies from back in the IGE days.
http://www.zam.com/story.html?story=25684&storypage=1ZAM: You’ve been hailed as something of a savior on the forums. How’s that make you feel?
ManWitDaPlan: I
find it fun in some ways - everyone loves to feel "special" after all -
but disconcerting in others - I'm not the spotlight-seeking type.
That
having been said, I do understand why some are making a big deal of it.
Trion sunk how many millions of dollars and years of time into making
Rift? Hint: Over $50 MILLION and at least a couple years. That is a LOT
of capital and work hanging out there. The last thing anyone at Trion
needs is to have the playerbase for a pay-to-play game lose confidence
in the game's internal security, so they set what I suspect is a new
speed record for fixing the exploit.
ZAM: This seems like something Trion should’ve found during
their QA testing. Why do you think it was missed? Was it something
really obscure? And how were you able to track it down when Trion
couldn’t?
ManWitDaPlan: I can't go into too many
details, but can say that the exploit would be easy to miss because
you'd have to be looking for something very specific in a very specific
place to find it. I found it because I was actively digging for it.
Trion was looking for it as well, according to what they and I had
discussed. I basically found it before they did.
ZAM: Do you still have confidence in the team? What’s their response to you been thus far?
ManWitDaPlan: Trion's
response to the revelation of the exploit has been spot-on. Steve
Chamberlin, the dev lead for Rift, was on the phone with me within five
minutes of my sending the technicals on the exploit, and while I was
talking to him, the engineering team was likely already editing and
recompiling code. A patch was deployed just over two hours after the
exploit was revealed. A few extra fixes (to Coin Lock) were also pushed
in at the same time to further tighten things up. The phrase "epic win"
is cliched from its overuse as a meme, but it nevertheless certainly
fits here.
Trion hit this like Jackie Chan channeling Bruce Lee,
which is what you do when you find an exploit. No playing the blame
game, no whining, just find and fix and slam the door on the hackers.
"Crush the hackers, see them driven from before you, and hear the
lamentation of their women!" (Apologies to Ahnold for that...)
ZAM: Do you feel comfortable with Trion’s response?
ManWitDaPlan:
Extremely so. The response was flawlessly executed, and should become a
textbook example of how a MMO company should respond to any
discovered bug - contact the person that found it, get the details,
verify their findings, act to secure the bug. Not only did the Trion
crew take the exploit seriously, they took fixing it seriously. I
mean, come on, reported discovery to implemented fixes in TWO HOURS?
I've never seen anyone in IT respond to bug reports that fast.
ZAM: There were a number of folks that helped you. Can you point them out?
ManWitDaPlan: TheScoo
was the hapless-but-willing victim of my tests once I locked down the
exploit's specifics. He allowed me to remotely access his account
(while he watched) and even let me delete a test character.
HomeFry helped me with some LAN tests and anti-malware scans on my
systems, and was on the network monitor while I was wrecking TheScoo's
characters and annoying Coin Lock with my escapades.
I bounced
some of the details I was seeing off the_real_seebs, who was also
looking into the hacking problem and came up with many of the same
conclusions I did. Basically I worked out a few key aspects of the
exploit before he did, so one way or another this mystery was gonna be
solved - if I hadn't gotten to the magic trick he surely would have.
ZAM: Are these sort of things common in MMOs, and do other companies simply keep it quiet?
ManWitDaPlan:
Security exploits can and do happen in any complex system. MMOs,
operating systems, you name it, the more complex the system the more
opportunities there are for something to go wrong. There are rootkits
for OSX and many Linux variants, Windows is notorious for security
issues (althogh that's slowing improving finally), the Stuxnet virus
targeted embedded systems in nuclear power plants, etc. etc. etc.
Security is fickle. It's finicky. It's nitpicky. It demands attention
to the minutae but will chastise those that cannot also see the big
picture. And it punishes the slightest mistake or miscue or omission
with the greatest severity.
Anyone that says
_insert_MMO_name_here_ is hackproof is delusional. Hacks exist for ALL
of them. To use a relevant example, WoW went to two-factor
authentication to stop the hacking it had since it launched, so the
hackers simply turned around and broke the algorithm that makes their
keyfobs for 2FA work. There's a lot of real money in selling virtual
things, and that means RMTers can afford to hire the best and brightest
of the bottom of the coding barrel. If there is a way to break a MMO,
there are people whose working time is devoted to finding it.
The million-dollar-a-month question isn't whether a vulnerability kept
quiet - no matter who you are and what you do, you never reveal an
exploitable weakness until after it's corrected - what makes the
difference is how it's handled once it's discovered. Trion wins
one-point-five Internets for their handling of this particular
nightmare.
ZAM: Does this change your outlook about Trion or RIFT at all?
ManWitDaPlan:
I have to admit, I was becoming increasingly concerned that the game
was broken to the point of being "unsafe" to play. By "unsafe," I mean
that there was the very real possibility that my account could be
wrecked at any time with no warning, so it would be a waste of time to
level up if I would end up standing naked and penniless next to a
mailbox the next time I logged in. So as I was posting my "eureka, found it!" post, I was hoping to be impressed by the response - if I wasn't, I was already planning to pull my subscription!
The response was insane. I wasn't contacted by some support flunky with
no authority to do anything but read from a script, I was on the phone
with the development team lead. I sent the technical details and got a
call back within minutes. Left work to head home, and by the time I
got home the server team lead was calling. Before I finished eating
dinner the exploit was fixed, some extra features were improved, and
all of Telara was cursing my name for making them go hunt down an
unlock code in their email.
If you're not the top dog you have
to fight the top dog to take his title. If you ARE the top dog that
title is yours to LOSE. If Trion can keep doing what they're doing when
it comes to gamer-centric behavior and a willingness to risk annoying
users during Friday night gaming in order to push through emergency
patches that make everyone's characters safer, and if they can overcome
the weaknesses this whole mess has exposed (such as having their
support team overwhelmed), the top dog had better be working on his
bite 'cause there's a new challenger to the throne.
ZAM: What do you think the future of the game is?
ManWitDaPlan:
Since Blizzard set the world on fire with World of Warcraft,
everything MMO-related since then has been compared to WoW and everyone
asks whether each newly emergent MMO is the "WoW killer." Rift stands
the best chance of dethroning WoW that I've seen of any contender to
date, and not strictly because the game brings something new/special to
the table. After all, Rift is derivative of all that went before it,
just as WoW was, and as Ultima Online was of MUDs/MOOs, etc. all the
way back to the first games writen for computers.
A key, and
often overlooked, part of the equation is how the game is run, how the
GMs interact with the players, how involved the developers are with the
playerbase, and whether the game's producer fosters a real sense of
community for and with their customers. After all, a game is only as
good as its developers make it and its players play it. Trion is
striving to do right on all counts, and that puts pressure on the whole
MMO world to do it better, whatever "it" might happen to be.
I
think this will ultimately mean a positive outlook for Rift, for
Trion, and for MMOs as both entertainment medium and creative artform.
The sky isn't even the limit; there's a lot of potential out there just
waiting to be tapped. Everybody wins.
ZAM: Is there anything else you’d like to say about this whole experience?
ManWitDaPlan: It's
been unusual to say the least, but thus far it's been a fun ride. Even
though this is definitely not a normal set of circumstances, if Rift
offers this much excitement - so much it spills over into meatworld - I
might have to play it for a while longer just to see what happens
next.
security exploit (that had nothing to do with ZAM) has been squashed.
Read our exclusive interview here!
For the past two weeks, the RIFT community
has been rocked by a seemingly endless onslaught of accounts being
compromised. Account security is an incredibly touchy subject with MMO
players, and once it became apparent that this was an epidemic rather
than a handful of occurrences, fansites like ZAM were blamed as part
of, as Trion put it, a “witch hunt” by some members of the community.
It turns out that fansite security had nothing to do with the whole mess.
The true culprit behind the account debacle
was tracked down by a member of the RIFT community that ignored all of
the speculation going on around him and put real effort into making
RIFT a safer place to play. Known only as ManWitDaPlan on the RIFT forums (and Webmaka on ZAM’s own forums), he exposed the flaw with the aid of several other committed RIFTers. Trion quickly found his post, contacted him directly, and within hours had the exploit fixed. The entire community, including all of us at ZAM, breathed a huge sigh of relief.
UPDATE: Trion speaks out and clears the air about ManWitDaPlan's find.
Over the last few hours, we’ve managed to reach out to MaWitDaPlan and
get his side of the story, find out what his background is, and see
what he thinks of the future of RIFT. We hope you enjoy this exclusive
interview!
ZAM: What’s your background? Do you often work with account security?
ManWitDaPlan: I've
been a programmer for most of my life (started at age ten), and
professionally so since 1995. I currently own a small security-software
company specializing in secure data destruction. I'm also something of a
"white hat" (aka "ethical hacker") in that I was involved with the
cracking and warez scenes in my younger days and have basically employed
the skillsets from that era of my life for more worthy pursuits.
ZAM: Why did you want to pin it down? Did you get an account hacked? Or did you just feel like doing a good deed?
ManWitDaPlan: My
account was hit during the start of the hack-fest on the weekend of the
12th-13th of March. I was left with only two pieces of armor and some
gold, but my bank and mailbox were completely untouched, which deviates
from the norm for what happens to hacked accounts. That made me
immediately suspect there was a bug of some sort.
Since I've been
working with security for so long I run a very, VERY locked-down system
and knew that there was pretty much no way I could have been hacked via
the usual avenues (e.g., malware). Still, I invoked some of my more
aggressive anti-malware scanners, including a couple that act as
hypervisors, and every scan I could throw at the systems turned up
nothing at present, and no signs of ever having been infected in the
past.
Between these two - the bank being untouched and the
systems being clean - I started looking up the chain from me to the game
servers proper. I started searching for man-in-the-middle attacks and
server-side compromises. That avenue began with investigating how the
game works, which led to finding an exploit, which led to working out
how it works, which led to the forum post that from the looks of things
shook all of Telara in a way that'd make Regulos go "umm, okay, let's go
find another planet to eat - these people are nuts!"
ZAM: There was a lot of finger pointing going on around the
community, especially towards ZAM RIFT and RIFT Junkies. What would you
tell those folks that were on, as Trion called it, a “witch hunt?”
ManWitDaPlan: Unfortunately
that sort of thing is normal human nature. 99.999% of the time a game
account gets hacked it was because the user of that account did
something dumb, like using weak passwords or reusing compromised
credentials, or just as frequently, allowing malware to get a foothold
and leech previously-safe credentials.
This time around, it was
an active, in-the-wild exploit, so the normal causes were not the
primary ones. Some people cannot remove the blinders of their own
preconceptions, though, and couldn't adjust to the idea that it wasn't
malware or poor client security in all cases, so they stuck with what
would be the most likely answer under normal circumstances.
Once it became increasingly clear there was more going on that just
bad/reused passwords and those folks were forced to rethink their "it's
got to be your fault" stance, the next target was anyone that ran
anything that could conceivably be a hacking vector. Thus, the finger
pointing toward fansites that offer any form of active content or
add-on. ZAM took an extra-tough dose of baseless blame thanks to the
old, no-longer-valid link to RMT companies from back in the IGE days.
http://www.zam.com/story.html?story=25684&storypage=1ZAM: You’ve been hailed as something of a savior on the forums. How’s that make you feel?
ManWitDaPlan: I
find it fun in some ways - everyone loves to feel "special" after all -
but disconcerting in others - I'm not the spotlight-seeking type.
That
having been said, I do understand why some are making a big deal of it.
Trion sunk how many millions of dollars and years of time into making
Rift? Hint: Over $50 MILLION and at least a couple years. That is a LOT
of capital and work hanging out there. The last thing anyone at Trion
needs is to have the playerbase for a pay-to-play game lose confidence
in the game's internal security, so they set what I suspect is a new
speed record for fixing the exploit.
ZAM: This seems like something Trion should’ve found during
their QA testing. Why do you think it was missed? Was it something
really obscure? And how were you able to track it down when Trion
couldn’t?
ManWitDaPlan: I can't go into too many
details, but can say that the exploit would be easy to miss because
you'd have to be looking for something very specific in a very specific
place to find it. I found it because I was actively digging for it.
Trion was looking for it as well, according to what they and I had
discussed. I basically found it before they did.
ZAM: Do you still have confidence in the team? What’s their response to you been thus far?
ManWitDaPlan: Trion's
response to the revelation of the exploit has been spot-on. Steve
Chamberlin, the dev lead for Rift, was on the phone with me within five
minutes of my sending the technicals on the exploit, and while I was
talking to him, the engineering team was likely already editing and
recompiling code. A patch was deployed just over two hours after the
exploit was revealed. A few extra fixes (to Coin Lock) were also pushed
in at the same time to further tighten things up. The phrase "epic win"
is cliched from its overuse as a meme, but it nevertheless certainly
fits here.
Trion hit this like Jackie Chan channeling Bruce Lee,
which is what you do when you find an exploit. No playing the blame
game, no whining, just find and fix and slam the door on the hackers.
"Crush the hackers, see them driven from before you, and hear the
lamentation of their women!" (Apologies to Ahnold for that...)
ZAM: Do you feel comfortable with Trion’s response?
ManWitDaPlan:
Extremely so. The response was flawlessly executed, and should become a
textbook example of how a MMO company should respond to any
discovered bug - contact the person that found it, get the details,
verify their findings, act to secure the bug. Not only did the Trion
crew take the exploit seriously, they took fixing it seriously. I
mean, come on, reported discovery to implemented fixes in TWO HOURS?
I've never seen anyone in IT respond to bug reports that fast.
ZAM: There were a number of folks that helped you. Can you point them out?
ManWitDaPlan: TheScoo
was the hapless-but-willing victim of my tests once I locked down the
exploit's specifics. He allowed me to remotely access his account
(while he watched) and even let me delete a test character.
HomeFry helped me with some LAN tests and anti-malware scans on my
systems, and was on the network monitor while I was wrecking TheScoo's
characters and annoying Coin Lock with my escapades.
I bounced
some of the details I was seeing off the_real_seebs, who was also
looking into the hacking problem and came up with many of the same
conclusions I did. Basically I worked out a few key aspects of the
exploit before he did, so one way or another this mystery was gonna be
solved - if I hadn't gotten to the magic trick he surely would have.
ZAM: Are these sort of things common in MMOs, and do other companies simply keep it quiet?
ManWitDaPlan:
Security exploits can and do happen in any complex system. MMOs,
operating systems, you name it, the more complex the system the more
opportunities there are for something to go wrong. There are rootkits
for OSX and many Linux variants, Windows is notorious for security
issues (althogh that's slowing improving finally), the Stuxnet virus
targeted embedded systems in nuclear power plants, etc. etc. etc.
Security is fickle. It's finicky. It's nitpicky. It demands attention
to the minutae but will chastise those that cannot also see the big
picture. And it punishes the slightest mistake or miscue or omission
with the greatest severity.
Anyone that says
_insert_MMO_name_here_ is hackproof is delusional. Hacks exist for ALL
of them. To use a relevant example, WoW went to two-factor
authentication to stop the hacking it had since it launched, so the
hackers simply turned around and broke the algorithm that makes their
keyfobs for 2FA work. There's a lot of real money in selling virtual
things, and that means RMTers can afford to hire the best and brightest
of the bottom of the coding barrel. If there is a way to break a MMO,
there are people whose working time is devoted to finding it.
The million-dollar-a-month question isn't whether a vulnerability kept
quiet - no matter who you are and what you do, you never reveal an
exploitable weakness until after it's corrected - what makes the
difference is how it's handled once it's discovered. Trion wins
one-point-five Internets for their handling of this particular
nightmare.
ZAM: Does this change your outlook about Trion or RIFT at all?
ManWitDaPlan:
I have to admit, I was becoming increasingly concerned that the game
was broken to the point of being "unsafe" to play. By "unsafe," I mean
that there was the very real possibility that my account could be
wrecked at any time with no warning, so it would be a waste of time to
level up if I would end up standing naked and penniless next to a
mailbox the next time I logged in. So as I was posting my "eureka, found it!" post, I was hoping to be impressed by the response - if I wasn't, I was already planning to pull my subscription!
The response was insane. I wasn't contacted by some support flunky with
no authority to do anything but read from a script, I was on the phone
with the development team lead. I sent the technical details and got a
call back within minutes. Left work to head home, and by the time I
got home the server team lead was calling. Before I finished eating
dinner the exploit was fixed, some extra features were improved, and
all of Telara was cursing my name for making them go hunt down an
unlock code in their email.
If you're not the top dog you have
to fight the top dog to take his title. If you ARE the top dog that
title is yours to LOSE. If Trion can keep doing what they're doing when
it comes to gamer-centric behavior and a willingness to risk annoying
users during Friday night gaming in order to push through emergency
patches that make everyone's characters safer, and if they can overcome
the weaknesses this whole mess has exposed (such as having their
support team overwhelmed), the top dog had better be working on his
bite 'cause there's a new challenger to the throne.
ZAM: What do you think the future of the game is?
ManWitDaPlan:
Since Blizzard set the world on fire with World of Warcraft,
everything MMO-related since then has been compared to WoW and everyone
asks whether each newly emergent MMO is the "WoW killer." Rift stands
the best chance of dethroning WoW that I've seen of any contender to
date, and not strictly because the game brings something new/special to
the table. After all, Rift is derivative of all that went before it,
just as WoW was, and as Ultima Online was of MUDs/MOOs, etc. all the
way back to the first games writen for computers.
A key, and
often overlooked, part of the equation is how the game is run, how the
GMs interact with the players, how involved the developers are with the
playerbase, and whether the game's producer fosters a real sense of
community for and with their customers. After all, a game is only as
good as its developers make it and its players play it. Trion is
striving to do right on all counts, and that puts pressure on the whole
MMO world to do it better, whatever "it" might happen to be.
I
think this will ultimately mean a positive outlook for Rift, for
Trion, and for MMOs as both entertainment medium and creative artform.
The sky isn't even the limit; there's a lot of potential out there just
waiting to be tapped. Everybody wins.
ZAM: Is there anything else you’d like to say about this whole experience?
ManWitDaPlan: It's
been unusual to say the least, but thus far it's been a fun ride. Even
though this is definitely not a normal set of circumstances, if Rift
offers this much excitement - so much it spills over into meatworld - I
might have to play it for a while longer just to see what happens
next.
