The LiDAR sensor on your iPhone could soon let you see around corners


Researchers at MIT Media Lab have found a genuinely jaw-dropping use for the LiDAR sensor sitting inside your iPhone and iPad Pro. It can detect and track objects that are completely outside the camera’s field of view. Yes, that means seeing around corners.

This kind of imaging, called non-line-of-sight (NLOS) imaging, is not a new concept. But past demonstrations relied on powerful, expensive lab-grade lasers with little application in the real world.

What makes this research exciting is that the MIT team pulled it off using the same low-power LiDAR sensor already embedded in our smartphones.

How does it work?

The team is using the LiDAR sensor to allow us to look beyond corners at objects that are not directly in our line of sight. The secret sauce is motion. As your device moves, the system simultaneously tracks the object’s shape, the object’s position, and the camera’s position over time.

The team calls this an aperture sampling model, and it essentially stitches together a series of noisy, imperfect readings into something meaningful. The outputs are not crisp photos of what is hiding around the corner. Instead, you get progressively richer inferences. The system can tell you something is there, how it is moving, and what shape it roughly has. Think of it like echolocation, but with light.

What can it actually do?

The team demonstrated four specific capabilities: tracking a single object, reconstructing its shape, tracking multiple objects at once, and something particularly interesting for robotics, which is camera self-localization using hidden landmarks.

That last one is a big deal. A robot or autonomous system that can orient itself using objects it cannot directly see has a massive advantage in the real world. It can also help improve the self-driving tech or delivery drones for things like accident avoidance.

Sadly, you cannot try this on your smartphone right now, “as that would require these companies to release their raw data, which they often don’t do,” said Siddharth Somasundaram, one of the researchers on this project. That said, the researchers have made their code publicly available, and the sensor hardware can be assembled for under $50.



Source link

Leave a Reply

Subscribe to Our Newsletter

Get our latest articles delivered straight to your inbox. No spam, we promise.

Recent Reviews


Modern displays are amazing when it comes to detail, brightness, color, and all the ingredients that make for an impressive picture—except motion clarity.

CRT screens are still the king of motion clarity, but plasma flat-panel screens hold a respectable second place, and in many ways I still miss my old 720p 51-inch plasma TV and the crisp motion I gave up by switching to a 4K LCD.

Plasma solved motion the “right” way

Plasma displays didn’t just show an image—they flashed it.

While they operate on different principles, CRTs and plasma TVs have a few things in common. First, the phosphors used by CRTs and plasma displays are the same. Second, because these phosphors fade quickly, they need to be continuously refreshed.

In a CRT, the electron beam scanning from the top to the bottom of the screen achieves this, and in a plasma, a high-speed electric pulse does the same. Because of this rapid pulse-and-fade, these screen technologies have crisp perceptual motion, since our brains tend to interpret moving images that don’t pulse as “smearing” across our retinas.

The pulsing nature of plasma technology isn’t the only reason for its better motion reproduction. These screens also have very low latency and very fast pixel response times. Combined, it’s not quite as good as CRT motion handling, but it’s significantly better than LCD and OLED technology, even today.

Modern TVs rely on sample-and-hold—and that’s the problem

Stand and deliver blurry images

Blur Busters UFO Test

Modern LCD and OLED televisions are “sample and hold” technologies. They can hold each frame of video perfectly for the entire duration of that frame without deviating in brightness and then instantly snap to the next frame without any dipping to black in-between.

On paper, this sounds like a good thing, but your eyes don’t stay still when tracking motion. As they follow a moving object, the image being held on screen effectively drags across your retina, creating the perception of blur. Even if the panel itself is perfectly sharp.

You might not even realize how blurry motion is on modern displays if all you’ve ever seen with the naked eye is an LCD or plasma. However, if you see a CRT or plasma in person, the difference is quite striking.

The sample and hold issue means that no matter how much you increase the refresh rate, that type of blur persists. It’s why my 85Hz CRT monitor is clearly less blurry in motion than my 240Hz LCD monitor. It’s especially apparent when you’re playing 2D games that scroll the entire screen, with LCDs or OLEDs smearing the image in a way that gives me a bit of a headache if I’m being honest.

Playing Diablo 2 on a CRT. Credit: Sydney Louw Butler/Shutterstock.com

It creates this weird situation where a modern TV can be incredibly sharp in a freeze frame but somehow look softer than a lower-resolution display that isn’t sample and hold as soon as you press play.

Motion interpolation is a workaround, not a solution

It’s an abomination, that’s what it is

One of the “fixes” that TV makers came up with to reduce unwanted motion blur is a technology known as frame interpolation, or more commonly “motion smoothing.” Here an algorithm creates fake frames that guess at what the middle step of motion would look like if it were captured. This creates a high frame-rate video output, which we see as smoother and more crisp.

While this doesn’t take away sample-and-hold blur, it does improve motion clarity. Unfortunately, it also destroys the intended frame rate that shows and movies were meant to be seen at. It’s also useless for video games, because it introduces an enormous amount of input lag. NVIDIA’s DLSS technology is also frame interpolation, but it works for games because of several mitigations NVIDIA put into the technology. These measures don’t exist on TVs.

While some people think motion smoothing isn’t all bad, TV makers are no longer activating it by default as much anymore, and my advice is to always turn it off because the trade-offs are just not worth it.

Screenshot 2025-07-01 at 9.21.03 AM

7/10

Brand

TCL

Display Size

85-inches

The 2025 model TCL QM6K Google TV delivers a stunningly clear and bright picture with a new Mini-LED panel, improved local dimming zones, Dolby Vision IQ, and a neat new Halo Control system for improved visuals. Get this TV and elevate your living room. 


Black frame insertion tries to recreate plasma—but comes with trade-offs

Who turned out the lights?

The other trick sample-and-hold screens have to mimic what CRTs and plasma TVs do naturally is called BFI, or Black Frame Insertion. As the name suggests, the display inserts a full black frame between every original frame. This provides an instant and dramatic increase in motion clarity. However, it also has a big impact on brightness. As much as half of the light is now gone, so the image is much dimmer. Pushing overall brightness to compensate makes things hotter and more energy-hungry.

Some BFI implementations cause visible flicker, for which I personally have no tolerance at all, but the biggest problem here is that BFI doesn’t have the smooth pulsing roll off of the phosphors used in CRTs and plasma.


The future might circle back—but we’re not there yet

That might be changing, however, because a new generation of LCDs can leverage the power of multi-zone backlight technology to strobe the backlight across the screen in a way that mimics a CRT scanline.

NVIDIA’s G-SYNC Pulsar has received rave reviews from the biggest motion blur haters, and I sincerely hope that a similar technology becomes standard in TVs going ahead, so we can go back to enjoying the crisp motion we used to have without all the compromises.



Source link