FAB: Fast and Accurate Body Estimation

in Pixels, World Space and Scenes

Jinkun Cao

www.jinkuncao.com

Teaser. Sources and licences: Credits.

Abstract

We present FAB (Fast and Accurate Body estimation), a human body estimator that achieves leading inference efficiency and body recovery faithfulness at the same time. As a 3D human pose estimator, FAB’s core is to predict human skeleton joints and surface dense points directly, rather than only parametric mesh model parameters. In pixels, FAB predicts dense 2D and 3D body points directly, alongside a parametric (MHR) head, and fits SMPL to them in one learned pass and two Gauss–Newton steps. Its ViT-L student FAB-L, distilled from a fine-tuned SAM 3D Body, runs in real time for one person on one RTX 4070 Ti SUPER, from person detection to a fitted body, 4.6× faster than SAM 3D Body’s body-only path in eager PyTorch, and is more accurate than SAM 3D Body on most camera-space benchmarks. The same network predicts heel, toe, hand, and knee contacts, which tie the body to the scene. In world space, FAB-L is more accurate than many recent methods. In scenes, FAB fuses the static scene’s depth around the visible person and aligns the scene and body through the predicted contacts. FAB also brings convenience for the robotics task of human-to-humanoid real-to-sim motion transfer. An in-model G1 head and FAB-Retarget, a motion-level retargeter that reads the dense points (or the mesh fitted to them), map the motion to a Unitree G1. FAB-Retarget’s references follow the person more closely than those of the specialized retargeters (SOMA, GMR, UMR, OmniRetarget) and much more closely than Fast SAM 3D Body’s body retargeted by our sequence IK.

Method

The FAB pipeline: one network per person and frame predicts dense points, 2D keypoints, body parts and heel / toe contacts, and the SMPL body is fitted to the points; per clip, the body is placed in a metric world, the scene is fused around it and the motion is retargeted from the dense points to a Unitree G1
One network per person and frame predicts dense points, keypoints, body parts and heel / toe contacts; SMPL is fitted to the points. Per clip: a metric, gravity-aligned world; the scene placed on the body via contacts; the motion retargeted to a Unitree G1 from the dense points (red frames). Walls cut above 0.5 m. Images: EMDB.

Images / Videos

Every output, every frame

FAB-L on every frame on its own (per-frame image model, no temporal smoothing), with the ground-truth box, on one crop around the person: 2D keypoints with heel / toe contacts (red: planted), body parts, the SMPL body fitted to the points and the dense 3D points (the side facing the camera). One re-run of the paper’s FAB-L on EMDB (per-clip MPJPE within 0.07 mm of the paper’s). Bystander faces blurred.

EMDB P8_69
EMDB P6_51
EMDB P6_47

Video and world

FAB-L’s world reconstruction from one moving camera: the scene fused from the video and coloured from its frames, the body in blue while it is in the image and grey when it is not, foot contacts from the video contact model as red dots, and the camera with its path. Top left: the input (bystanders blurred).

Picked by our WA-MPJPE gain over GVHMR on the 25 EMDB-2 sequences: the single-method clips are P3_28 (8th of 25) and P9_79 (19th), the nearest passes to one third and two thirds; the 6 smallest gains (1st–6th, including the two where GVHMR is better) and the 6 largest (20th–25th) failed our scene check (planted feet within ±5 cm of the reconstructed scene) in their 8 s windows. Comparisons below: P3_29 (14th of 25), P3_27 (6th), the RICH-191 clip nearest the median gain, and P9_79 (19th). On EgoBody the floor under the feet is only partly reconstructed, and the planted feet sit about 7 cm above the floor around them.

Ours | DuoMo | GVHMR

human visible not in image foot contact · Left: ours, then the baselines (released code). A body is blue while at least half of the ground-truth body’s joints are in the image and grey otherwise, by the same rule for every method.

Scenes

FAB fuses the static scene’s depth around the visible person and aligns scene and body through the predicted contacts. On the same videos as VideoMimic’s released real2sim pipeline, our scene is more accurate per frame on RICH, and, placed by the ground-truth camera, our body lies closer to the scanned contact surfaces. All four methods ran on the same video, with no tuning on this clip.

EMDB P3_30

Humanoid

Each retargeter’s Unitree G1 reference, from the same world body, tracked by the same released tracker (SONIC, G1 mode) in one MuJoCo setup. Ours, points / mesh: FAB-Retarget in points / mesh mode, on the dense points and the SMPL fitted to them / on the fitted SMPL alone. GMR: input turned to face the robot’s initial heading at frame 0. Silver: the tracked G1; blue: the method’s kinematic reference, drawn at the robot’s position. bone err is the running mean of the G1’s bone-direction error against the real person up to its first fall; root lag is the current horizontal distance between the robot’s root and its reference root; the red badge marks a fall. Left: the input with FAB-L’s dense points and the mesh fitted to them, and the video contact model’s feet on the ground.

Clips picked by rule: the largest, median and smallest gain of Ours, mesh over Fast SAM 3D Body (SONIC-G1 bone-direction error) over the 12 trackable test clips; all three are EMDB clips. More EMDB clips in the other tabs.

From one video to a humanoid: in-the-wild Pexels clips, selected examples.

EMDB P5_44

Many people, one world

Group dances from stock video: every dancer through FAB, all of them in one metric world with the scene, each retargeted to a Unitree G1 and tracked in simulation. Selected examples; robots are retargeted from the fitted bodies and tracked independently, trackers are named in each video (cheerleaders: SuperMimic, one tracker trained per dancer, in a world levelled by one rotation); bystander faces blurred.

Pexels 7894075 (MART PRODUCTION)
Pexels 8688883 (ANTONI SHKRABA production)

Ego + exo: body and hands in one world

One person seen by two cameras: FAB on a third-person camera, FAH-L on the camera the person wears on the head, synchronised through the head camera’s timecode, in one world. Hands come from our companion model FAH (project page, code). Robots are FAB-Retarget references (kinematic); hands are shown in the head-mounted view only; floors are not levelled; bystander faces blurred.

Key numbers

39.2 fpsFAB-L end to end on one RTX 4070 Ti SUPER, person detection included; SAM 3D Body’s body-only path: 8.5 fps (4.6×)
32.7 mmPA-MPJPE on 3DPW (SAM 3D Body: 33.7 mm)
57.1 mmWA-MPJPE on EMDB-2, all 25 sequences; the best re-run world method, DuoMo: 67.5 mm
50.6 mmWA-MPJPE on RICH-191 with ground-truth boxes, as every re-run world method but CRISP is given
8.9 cmbody to the scanned contact surfaces on RICH, placed by the ground-truth camera (VideoMimic: 23.8 cm)
11.8°G1 reference bone-direction error, FAB-Retarget [skeleton + mesh + points] (GMR: 14.7°)

From the paper’s tables, cell by cell; our rows shaded, bold: best.

Camera space

3DPWRICH
MethodPAMPJPEPAMPJPEfps
TRAM36.4†58.840.068.71.0
GVHMR38.4◊†58.2◊38.066.69.2
GEM-X (GENMO)37.659.835.356.42.0
WHAM38.3†63.842.277.92.8
DuoMo36.7†59.435.049.51.1‖
SAM 3D Body33.7†55.131.350.08.5‡
Fast SAM 3D Body34.1†55.531.449.86.7
NLF-L35.7†57.728.1†45.310.8
PromptHMR37.1†59.137.154.57.9
FAB-H, MHR mesh → SMPL, official32.4†52.727.9†46.00.12
FAB-H, MHR mesh → SMPL, our fit33.1†53.228.2†45.624.9
FAB-H, dense points → SMPL33.7†54.829.8†48.221.8
FAB-L (ours)32.7†53.127.8†45.539.2
FAB-S (ours)37.7†59.835.4†56.463.8

Camera space, re-run by us under the standard protocol on the same frames (mm; PA: PA-MPJPE; 3DPW: 14 H36M joints; RICH test: 24 SMPL joints). FAB-L / FAB-S: SMPL fitted to their dense points; FAB-H is also reported with its MHR mesh converted to SMPL, by our fit and by MHR’s official converter (official). Bold: best per column; shaded: ours (also in the other tables). fps: end to end on one RTX 4070 Ti SUPER. †Trains on this benchmark’s training split (NLF-L on RICH: split unstated). ◊Flip test-time augmentation. ‡Body-only mode, eager PyTorch. ‖Exceeds the card’s 16 GiB.

World space

EMDB-2 (25 sequences)RICH-191
MethodBoxCam.WAWRTEJFSCam.WAWRTE
TRAMGTGT K, SLAM93.3328.33.2018.613.9GT K, static95.9154.52.24
GVHMRGTest. K, DPVO109.1272.81.9016.13.4GT K, static77.1125.12.34
GEM-X (GENMO)GTGT K, GT R114.2270.92.5435.34.3GT K, GT R79.3128.52.58
WHAMGT*FOV, GT R148.8384.54.6322.05.9FOV, static117.5201.95.06
DuoMoGTGT K, TRAM SLAM67.5167.91.199.53.9GT K, static55.683.41.70
CRISP§ (human–scene)ownMegaSaM74.7208.71.4653.612.4MegaSaM91.7158.13.37
SAM 3D Body56.0141.01.065.26.373.9144.05.67
Fast SAM 3D Body56.3141.11.065.36.474.4145.45.70
NLF-L60.4151.81.145.76.969.7†137.25.72
FAB-H, MHR mesh → SMPL55.4142.51.095.26.268.8†135.55.75
FAB-H, dense points → SMPL59.0147.31.045.57.775.6†150.26.01
FAB-L (ours)57.1147.21.065.36.865.9†132.25.73
FAB-S (ours)61.9163.81.135.88.680.1†155.25.93
FAB-L: ViPE camera at its own scale, smoothedoursViPE scale60.9170.41.554.77.6ours65.9†132.25.73
FAB-L, ground-truth camera motionoursours K, GT motion51.0137.90.194.06.8ours K, GT motion65.9†132.55.73
FAB-L (ours), ground-truth boxesGTours55.7143.01.035.16.2ours50.6†77.41.16

World space on the same frames: EMDB-2 (all 25 sequences) and RICH-191. Top: world methods with their released code and own inputs. Middle: per-frame bodies through our world pipeline, with the same cameras, boxes and contacts. Bottom: before the joint human–scene step, and with the ground-truth camera motion; last row: FAB-L with the ground-truth boxes. WA / W: WA- / W-MPJPE (mm); RTE (%); jerk J (10 m s-3); foot sliding FS (mm). Box: GT, ours (our detector) or own (its masks). Cam.: intrinsics K (FOV: fixed field of view) and camera motion (own SLAM / DPVO / MegaSaM, GT rotation R, or static); ours: our focal rule and ViPE. Bold: best in the top and middle blocks and the last row. *RICH-191: box from WHAM’s ViTPose keypoints in the GT box. §Overlapping 600-frame chunks. †Trains on RICH’s training split (NLF-L: split unstated).

Foot contact

staticmatHSC
MethodEMDB-2RICHMOYORICH
keypoint height / speed rule0.460.620.550.65
WHAM (released code)0.630.870.660.91
GVHMR (released code)0.710.890.610.90
ours, image head0.670.880.890.97
ours, causal video model (world frame)0.690.890.870.97

Foot contact F1 against three label sets. Static: world-fixed contact from the ground-truth world motion; mat: MOYO’s measured pressure mat; HSC: any contact against RICH’s human–scene contact annotations; all on validation splits, which share subjects with training. WHAM and GVHMR output only static contact, scored against every label set.

Scenes

scene geometryhuman in the scene’s worldhuman–scene contact
RICH-3EgoBodySLOPER4DPROXSLOPER4DEMDB P3_30RICH-3¶VideoMimic (123), G1
MethodrigidSim(3)rigidSim(3)ATEscene1-way2-waynon-pen.↑WAWRTEWAWpen.float|s|feetpen. >2 cmgap
VideoMimic18.411.710.97.63.049.824.631.084.835.773.811.410.836.510.213.623.84.357.25.0
CRISP8.16.97.66.12.475.8§18.627.376.018.1§67.2§3.6§6.4§14.9§7.511.619.26.50.8‡1.9‡
JOSH25.315.222.26.99.9264.674.544.287.322.277.74.640.883.24.94.19.035.40.7‡2.6‡
FAB-L (ours)†8.07.512.59.42.294.421.317.892.010.640.81.15.914.33.55.48.94.8**0.62.5

Scenes against three released pipelines, run by us on the same videos, under VideoMimic’s protocols where it has them; one measure per column, grouped by dataset. Units: cm, except non-pen., RTE and pen. >2 cm (%); lower is better except non-pen. §Overlapping 600-frame chunks. **After the joint human–scene step (10.5 before it). ¶pen. / float / |s|: camera-frame bodies placed in the scan by the ground-truth camera. †Trains on RICH’s and EgoBody’s training splits. ‡CRISP / JOSH: bodies and scenes through FAB-Retarget’s mesh mode, as ours; JOSH, which outputs no gravity, uses ours.

Humanoid

referencetracked
Referenceboneslipfalls, bone
SOMA15.50.285, 23.4
GMR14.70.265, 19.6
UMR18.20.395, 24.9
OmniRetarget21.70.318, 27.4
sequence IK24.30.226, 25.8
Fast SAM 3D Body25.50.396, 27.6
FAB-L G1 head (per frame)14.50.385, 20.4
FAB-Retarget [skeleton]11.70.125, 19.2
FAB-Retarget [skeleton + mesh]11.60.116, 20.0
FAB-Retarget [skeleton + mesh + points]11.80.134, 18.3

G1 references on 25 flat-floor test clips, tracked by SONIC (G1 mode) in one MuJoCo setup, all from the world pipeline’s body. Reference: bone-direction error (°) and sole slip (m s-1); tracked: falls of 25, bone-direction error before the first fall. FAB-L G1 head (per frame): the in-model G1 head; FAB-Retarget [skeleton] / [skeleton + mesh] / [skeleton + mesh + points]: its skeleton / mesh / points mode, pinned in one batch; Fast SAM 3D Body: its own SMPL through our sequence IK.

BibTeX

@misc{fab2026,
  title         = {Fast and Accurate Body Estimation in Pixels, World Space and Scenes},
  author        = {Cao, Jinkun},
  year          = {2026},
  eprint        = {2611.25801},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.25801}
}

Credits