Apple's trackball Apple Pencil patent: Drawing on walls just became possible

Apple just dropped a patent that could completely change how we think about the Apple Pencil. We're talking about a stylus that doesn't need an iPad screen—or really any screen at all. You could sketch on your desk, write on the wall, or even draw in mid-air. Let's break it down.

What you need to know:

• Surface-free operation: The new Apple Pencil could work on any surface—or even hovering above it—using built-in optical sensors • Trackball design: One version features a trackball tip that rolls as you move, tracked by internal optical sensors instead of mechanical parts • Universal compatibility: Patent suggests it could work with iPhone, MacBook Pro, Apple Watch, and more—not just iPad • Beyond drawing: The patent lists 20+ functions including security access, cursor control, and 3D object creation • Detection range: Works up to 100mm (nearly 4 inches) above surfaces, enabling true air-drawing capabilities

A newly granted patent suggests that Apple is working on a stylus that doesn't need a screen or even a surface to function. The U.S. Patent and Trademark Office just granted Apple U.S. Patent No. 12,353,649, suggesting the company is thinking well beyond the iPad when it comes to the future of the Apple Pencil.

Here's what makes this patent so game-changing: According to Patently Apple, this next-gen Apple Pencil could let you write or sketch in the air or on pretty much any surface — like a wooden table or even your wall — and still track your input with precision.

This isn't just wishful thinking—we're looking at some seriously clever tech that could make the Apple Pencil work with your iPhone, MacBook Pro, or literally any surface you can think of.

How optical sensors make magic happen

The heart of this patent lies in optical sensor technology that sounds like something out of sci-fi. The idea is to use optical sensors inside the stylus that can track movement, tilt, and position in 3D space. The patent even says this could work from a short distance, such as 10 centimeters above a surface.

Apple's patent describes two main optical sensor types: optical flow sensors and laser speckle flow sensors. Optical flow sensors could detect spatial-temporal image brightness variation (e.g., frame-to-frame brightness variation) to estimate motion of the input device. Meanwhile, laser speckle flow sensors could detect speckle patterns resulting from coherent illumination of an imaging configuration to estimate motion of the input device.

What makes this especially impressive is the range and versatility. According to Apple, both systems can detect movement even when hovering above a surface (up to 100mm, or 3.94 inches), while also allowing the stylus to operate on non-touch-sensitive materials like wood or paper. This 100mm hover distance means you could sketch preliminary ideas in the air before committing to the surface, or work on transparent materials like glass where the sensors detect movement above rather than through the material.

The trackball twist that changes everything

Here's where things get really interesting. One version of the Apple Pencil described in the patent uses a trackball at the tip — think old-school computer mice — but instead of mechanical tracking, Apple is suggesting using internal optical sensors to detect the ball's motion. One version of the Pencil described in the patent features a trackball tip, which rolls as you move the stylus.

According to Apple, this design also improves signal-to-noise ratios and works across more types of surfaces, regardless of their texture or reflectiveness. In user terms, this means better accuracy and reliability whether you're working on smooth glass, textured paper, or even rough wood surfaces.

The patent says optical sensor(s) can be configured to measure displacement of internal components of a stylus, allowing it to work even when not in direct contact with a surface. The trackball design also opens up possibilities for texture-sensitive input—imagine the pencil detecting whether you're drawing on rough paper versus smooth glass and adjusting line weight accordingly.

Beyond drawing: A universal input device

This isn't just about making bigger canvases for your art projects. The practical applications get really exciting when you look at the broader picture. Picture using the Pencil to sign documents on your MacBook trackpad, navigation gestures on your iPhone's back, or even authentication by drawing your signature in the air.

Apple's patent hints at combining this motion-sensing tech with other sensors, like force detectors, inertial measurement units, and even machine vision. In simple terms, that means the Apple Pencil could eventually support air gestures, better handwriting recognition, or even 3D object creation.

The patent has been granted a new patent that could radically change how the Apple Pencil works, allowing it to be used with any iPhone, MacBook Pro, or even Apple Watch. Apple even lists 20 or more functions such an Apple Pencil could perform that have nothing to do with drawing. Significantly, Apple proposes that such an Apple Pencil could be used to permit "authorized individuals access to restricted areas of the computer or computer network."

PRO TIP: Current Apple Pencil users should know this tech might not be backward-compatible—the optical sensors would likely require entirely new hardware.

What this means for iPad users

Look, patents don't always become products, but this one feels different. As someone who's hit the edge of an iPad screen mid-sketch more times than I can count, the freedom to work on any surface sounds genuinely liberating. The key to Apple's idea is that all the sensors needed for the Apple Pencil to function would be integrated into the accessory. Apple's patent stresses that this isn't meant to replace touch-sensitive displays, but it frees the Apple Pencil from the limitation of that 11-inch or 13-inch iPad screen.

For digital artists, this could mean sketching concepts on physical paper during meetings, then seamlessly continuing the work on their iPad—with the Pencil bridging both worlds. A future version of the Apple Pencil might feature optical sensors that could allow it to generate content on any surface. The patent describes sketching on any surface, making it possible to work on your desk while the Apple Pencil tracks the movements and transmits the data to the device. This could make the Apple Pencil less of an iPad accessory and more of a universal input device that works across Apple's entire ecosystem.

Where do we go from here?

Real talk: this patent is pretty ambitious, even by Apple standards. But the technical hurdles—like maintaining precision on uneven surfaces or handling varied lighting conditions—are exactly the kind of problems Apple's optical sensing team has tackled before with Face ID and LiDAR. The lead inventor listed on Apple's granted patent is William Whitney, Engineering Manager, Optical Sensing, which suggests serious engineering talent is behind this.

The patent is credited to five inventors, showing this isn't just one person's wild idea. For such a seemingly specific topic, the patent is surprisingly broad in its range of proposals.

Will we see this in the next Apple Pencil? Hard to say. But the fact that Apple is thinking this big about stylus input suggests they're not done revolutionizing how we interact with our devices. And honestly? Drawing on walls with precision tracking sounds pretty cool to me.