In 2010, when Microsoft unveiled the Kinect, it pitched the camera as a revolutionary new gaming device. Swing an imaginary lightsaber and that would be translated onscreen. Throw a football and it would be caught on your TV. Fifteen years later, we know the Kinect as an expensive failure. Microsoft overestimated the demand for playing games with your body. But the Kinect did still turn out to be revolutionary — just not for gaming.
Now, we understand the Kinect is anything but a gaming device. It became a robotics game changer, enjoyed a brief dalliance with pornography, and is now upsold as a ghost hunting toy. None of which would have been possible had a community of hackers not come together to fashion open source drivers for the Kinect, freeing it from the limitations of being locked to the Xbox 360 and opening new frontiers of experimentation, creative expression, and commercial advancement.
“Technically, nothing the Kinect did was entirely new,” says Memo Akten, an artist working with code, data, and AI and an assistant professor at the University of California, San Diego. The small camera projected a grid of infrared dots and read deformities in that pattern to discern depth. In an early example of machine learning, it recognized human limbs and gestures. “Those capabilities existed in research and industrial systems for many years,” he adds. Those systems cost in the region of $5,000 to $12,000. Here was Microsoft selling a variation of the technology for $150.
“What had previously required very expensive equipment and/or complex multi-camera setups with manual alignments, calibration, and correspondence was now available off the shelf,” Akten continues.
Kyle Machulis, CEO of Nonpolynomial and founder of buttplug.io — an open source project for controlling sex toys — was working on $250,000 mapping systems not dissimilar to the Kinect in 2010. He quickly recognized the peripheral as an opportunity to “democratize that technology.”
He recalls heading out on November 4th to pick up a Kinect to reverse engineer. An hour later, New York-based DIY electronics producer Adafruit announced OpenKinect: a bounty of $1,000 — a prize that it would raise to $3,000 — for whomever offered evidence of the Kinect working on any operating system.
“Imagine being able to use this off the shelf camera for Xbox for Mac, Linux, Win, embedded systems, robotics, etc.,” Adafruit wrote in its announcement. “We know Microsoft isn’t developing this device for FIRST Robotics, but we could! Let’s reverse engineer this together, get the RGB and distance out of it and make cool stuff!”
Doing so was not a simple case of taking the Kinect apart or plugging it in. Though it could connect to a PC via USB, the way they communicated was unknown and the only way to get at it was to watch the Kinect and Xbox 360 speaking to one another. “Since the Kinect didn’t have PC drivers, we needed this piece of hardware called a USB sniffer,” Machulis tells The Verge. A colloquial term for a protocol analyzer, a USB sniffer is a tool that could record the data passed between the Kinect and Xbox 360. In 2010, that cost $1,200 and, Machulis says, “I really didn’t want to buy it.”
Some information could be gleaned by simply connecting the Kinect to the PC, but it was mostly unhelpful — power consumption, packet sizes, and confirming the Kinect is, in fact, a camera. Hackers could start sending random packets and possibly work something out, but it was just as liable to brick the Kinect completely.
Hackers and reverse engineers around the world were raring to go. But it appeared that whoever got their hands on a sniffer would win the bounty almost by default. That race wasn’t just for the money, however, but also the cachet of being the first to hack such a high-profile device. With the community stalled over the massive expense — almost half the bounty — it opened the door for someone outside the community to potentially snatch the glory away.
To keep the contest equitable — and, perhaps, to try and maintain the bounty’s and the company’s momentum in the press, Adafruit took on that expense, ordering a sniffer to then release the logs to the community. But while everyone waited for the device to ship to Brooklyn, it appeared the worst had already happened and someone had beaten them to the punch.
“AlexP” released a video the next day demonstrating control of the Kinect’s motor on PC. It prompted a panicked response from Microsoft denying the Kinect could be hacked while threatening to explore legal options. Microsoft quickly U-turned when it became clear no one was trying to hack consumers’ cameras. But as the community reeled from potential litigation, AlexP returned with a second video. This time, he showed off depth and RGB images on PC. The contest was over before it had begun.
AlexP didn’t claim the bounty, however. Instead, his company, Code Laboratories, opened a $10,000 “fund,” upon payment of which Code Laboratories would release the source code to the open source community. For some in the OpenKinect community, this wasn’t so far removed from what Adafruit — which was already benefitting from significant press — was doing. Even if Adafruit was looking to open source the Kinect drivers and Code Laboratories to sell the drivers as it had with the PlayStation Eye before. For others, it was tantamount to a ransom, withholding code that could make the Kinect more accessible and unleashing its potential easier. “But that was great motivation for the community to just be like: Let’s take $10,000 away from you, actually,” Machulis says.
A competitive edge shouldn’t be surprising in a contest for a bounty. According to Machulis, however, there was more to it. “That’s the thing about reverse engineering; It’s who gets their name on it first and loudest.”
On the evening of November 9th, Adafruit finally uploaded the logs collected by its sniffer, and the community began to pore over them. They were searching for the protocols that controlled the Kinect, exploring packets that might turn on a light, enable a camera, or operate the motor. It was incremental, tedious, and exhausting work.
As US hackers dropped off in the early hours, the clock ticked over to the Kinect’s European release date. Like his American peers, 20-year-old Hector “marcan” Martin purchased a Kinect and, armed with Adafruit’s logs, went through packet by packet to divine the Kinect’s protocols.
When the US woke, it did so to the fruits of his dogged examination: a video of Martin demonstrating RGB and depth on Linux. It had taken six days for OpenKinect to hack the Kinect from its release — really, once the logs became available, it had taken Martin less than 24 hours.
That was far from the end, however. “Hector certainly did most of the hard, technical work in terms of getting the initial packets set up,” Machulis says. “Then everyone realized this shit is gonna get big.” The bounty claimed, the community set its sights on more drivers.
Theo Watson spent three weeks at his computer working on OpenKinect — every day, 10 hours a day. The Kinect revolutionized how Design I/O, which Watson co-founded, developed interactive installations, and he still uses it today. In 2010, however, he was 30 years old, recently transplanted to the US, isolating limb data from infrared cameras.
“I saw that time off as an R&D investment,” he says, a way to open more efficient avenues to bigger and better interactive experiences. “I really wanted to be the first person to get the Kinect running on a Mac.”
The OpenKinect community had coalesced around a remote nucleus of reverse engineers from different countries, generations, and demographic groups. There was drama, frayed nerves, but also a common goal, as Watson discovered while trawling through the process of running the Kinect on OSX.
“I think it really helped because it felt like you were part of a team effort,” he says. “If people were running into problems, it’s like having a collective brain. The Kinect basically needs everything to be perfect. If you’re off by one little thing, you don’t get anything. Then, suddenly, someone notices something and it works.”
Driven by the community, with a bit of help from Martin, Watson had the Kinect talking to Mac by November 12th. More drivers and code filled the community’s GitHub, and hacking efforts wound down. “There’s only so much to extract and then you have to be able to do something with that data,” Machulis says. “Those are two fairly different skillsets.”
Instead, people like Akten stepped in. The first open source drivers, libfreenect, didn’t include body tracking — Microsoft released its own skeletal SDK in 2011, its hand forced by OpenKinect — only granting access to raw depth data. “We could still do a lot with that,” Akten says. “For one thing, we had the 3D data, which allowed all kinds of creative, playful interpretations.”
The community chat flooded with experiments, many Akten’s. He explored drawing in 3D space, later moving on to develop machine learning algorithms to detect poses, and even controlling drones. The Kinect was open, bringing with it a host of explorations of how to exploit it creatively. Suddenly, an affordable way for robots to detect obstacles and map environments in real time became available, surgeons explored examining scans contactless, rapid 3D models of rooms and objects became a real possibility, teachers used the Kinect as an interactive learning device, and, if you really wanted, someone could now control a sex toy over a video call.
“This thing on the front camera,” Watson says, pointing to the black bar at the top of his iPhone’s screen, “that, I think, is a miniature Kinect.”
He’s almost wistful. Apple purchased PrimeSense, the Israeli company behind the Kinect’s sensor technology, in 2013. “I was so disappointed,” he says, “because I just knew that was the end of the Kinect technology.”
The sale prompted Microsoft to explore a new system for its next Kinect — OpenKinect went and hacked that one too — discontinuing Kinect for Windows shortly after its release in 2014 and shutting down manufacturing for the Kinect in 2017 as sales diminished and it focused on the Kinect 2 and development of the third-generation Kinect Azure. Yet, the technology has lived on, incorporated into countless Apple devices as part of its facial recognition and 3D mapping to the point of being ubiquitous.
That sense of loss extends, in part, to the internet from which OpenKinect emerged. “It was way more punk rock!” Watson laughs. “No one had really established the rules.”
In 2010, the internet was unruly; it had yet to coalesce around the hubs it has today. Piracy was in its heyday, pre-AlexNet — a major neural network architecture that paved the way for modern AI models like Stable Diffusion — with GitHub, now an online staple, having released only three years before (the same year as Tumblr and the iPhone’s reveal). “We were only four or five years into the maker movement,” Machulis says. “The idea of a product like this that has taken a massive amount of R&D cost to be put out and hacked this quickly — it was basically unheard of.”
Now, with better tools, it’s far more common. Which is part of why we don’t hear about it as much as before — that, and not being attached to, as Machulis puts it, a “shining sun” of a product. “It is in general easier to make some of this stuff,” Machulis continues. “There’s way more communities online, there’s more content creators talking about this stuff.” The kind of effort surrounding opening the Kinect has now lost some of its buccaneering flavor, some of its sense of counterculture, simply by virtue of becoming more mainstream and, in many ways, more frequent. “I don’t think anything fizzled out,” Machulis adds. “I think it just got quieter and spread out.”
Still, there is a sense that how we approach technology has changed irreparably. “I think technology has become more of a product now and less something that you get involved with. That’s kind of sad,” Watson adds. “I kind of fear that the current generation is growing up just thinking the internet is inflexible. It is the way it is and nothing will ever change. We were constantly surrounded by that change. And it really made things feel more free and more open.”
Rather, similar communities to OpenKinect may feel invisible without a subject as high-profile as the Kinect. As the economic bubble inflating around AI grows more opaque as corporate interests scramble to make the technology a profit-turning industry, hackers have turned their attention to open sourcing its models. Aligned to a sprawling technology constantly in the public eye, it may well be that AI grants us our next big communal reverse engineering effort to echo OpenKinect. This is especially curious given the Kinect gave many of us our first interactions with AI, and it is AI that is finally interrupting the Kinect’s spirited afterlife.
“When the original Kinect came out, it took what might have been 100 hours of me writing computer vision code with a standard black-and-white infrared camera and gave me something that would shave that time off our development for a project and give better-quality results,” Watson says. “AI with code is doing a similar thing; they just take away the painful aspects of the work and let us focus on the creative part.”
Now, 15 years after hackers opened the Kinect to computer vision creatives, AI can do everything it did better, faster, and using standard RGB cameras. Watson shows The Verge a video of AI’s real-time tracking, its superior occlusion of limbs and digits blitzing across the screen as members of a K-pop group weave around one another, each marked by a colored skeleton on the screen — all pulled from an ordinary camera. “AI is made to make decisions about many things very quickly, and we need a decision about every pixel in an image,” Machulis says. “Since we can tell so much just from images now we may not need all the extra hardware, with methods like gaussian splatting we’re already seeing that ability to, what looks like, create information from thin air.”
“Next time we chat, we might have gone back to infrared cameras,” Watson says, before adding: “AI might kill the Kinect.”
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
- GamingClose
Gaming
Posts from this topic will be added to your daily email digest and your homepage feed.
FollowFollow
See All Gaming
- TechClose
Tech
Posts from this topic will be added to your daily email digest and your homepage feed.
FollowFollow
See All Tech
