Update FMPose3D modelzoo integration#3221
Update FMPose3D modelzoo integration#3221deruyter92 wants to merge 19 commits intoDeepLabCut:mainfrom
Conversation
- Add fmpose3d_humans and fmpose3d_animals framework mapping to pytorch.
- Branch early in video_inference_superanimal(...) when model_name.startswith("fmpose3d").
- Route directly to _video_inference_fmpose3d(...)
Add _video_inference_fmpose3d(...) as a dedicated path.
Keep batch loop minimal in DLC:
- result_2d = api.prepare_2d(np.stack(frames))
- result_3d = api.pose_3d(result_2d.keypoints, result_2d.image_size)
DLC responsibilities only:
- iterate video frames/chunks,
- convert 2D results to DLC {"bodyparts": ...} layout,
- save .h5, .json, _3d.json, and optional labeled video.
deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py
Outdated
Show resolved
Hide resolved
C-Achard
left a comment
There was a problem hiding this comment.
Thanks, looks good overall. I generally agree with your point on separation of concerns, but probably the main concern for now is that it conforms to the modelzoo API
| import deeplabcut | ||
|
|
||
| video_path = "demo-video.mp4" | ||
| deeplabcut.video_inference_superanimal( | ||
| videos=[video_path], | ||
| superanimal_name="superanimal_quadruped", # ignored for fmpose3d models | ||
| model_name="fmpose3d_animals", | ||
| batch_size=8, | ||
| fmpose_return_3d=True, # include 3D dataframe in returned payload | ||
| ) |
There was a problem hiding this comment.
I think being able to call it like this is really nice in the end, great job !
There was a problem hiding this comment.
Pull request overview
This PR integrates FMPose3D monocular 3D pose estimation into the DeepLabCut modelzoo pipeline by creating a dedicated inference path. It relocates FMPose3D functionality from deeplabcut/modelzoo/fmpose_3d to deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d and routes FMPose3D models through the video_inference_superanimal function with early branching. The PR standardizes outputs by writing both 2D predictions and 3D artifacts (*.h5 and *.json files) and provides an optional return flag for in-memory 3D DataFrames.
Changes:
- Relocated FMPose3D wrappers from
modelzoo/fmpose_3dtopose_estimation_pytorch/modelzoo/fmpose_3dwith enhanced inference pipeline - Added early routing in
video_inference_superanimalforfmpose3d_*models with dedicated inference function - Removed preliminary FMPose3D recipe notebook and updated documentation to include usage example in ModelZoo.md
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/pose_estimation_pytorch/modelzoo/test_fmpose_integration.py | Updated imports and added comprehensive unit tests with new pytest markers (unittest/functional) |
| pyproject.toml | Added pytest markers for fmpose3d, unittest, and functional test categories |
| docs/recipes/fmpose3d.ipynb | Deleted preliminary recipe notebook (reverting PR #3208) |
| docs/ModelZoo.md | Added FMPose3D usage example with video_inference_superanimal API |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py | New dedicated FMPose3D inference pipeline with 2D-to-3D conversion and dataframe generation |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/fmpose3d.py | Relocated FMPose3D API wrapper with model metadata definitions |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/init.py | Added module with refactoring markers for future keypoint/pandas migrations |
| deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/README.md | Added brief FMPose3D overview documentation |
| deeplabcut/modelzoo/video_inference.py | Added early branching logic for fmpose3d models with parameter forwarding |
| deeplabcut/modelzoo/models_to_framework.json | Registered fmpose3d_humans and fmpose3d_animals as pytorch models |
| deeplabcut/modelzoo/fmpose_3d/fmpose3d.py | Deleted old location of FMPose3D wrapper |
| deeplabcut/modelzoo/fmpose_3d/init.py | Deleted old module initialization |
| _toc.yml | Removed fmpose3d recipe from table of contents |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
deeplabcut/pose_estimation_pytorch/modelzoo/fmpose_3d/inference.py
Outdated
Show resolved
Hide resolved
|
< Note: this PR depends on the bleeding-edge code from the FMPose3D repository > Shall I make a new release of FMPose3D on pypi? |
|
Here I disagree:
Because it uses the 2D SA_quadruped data, or human data, for example, then the architecture is the 3D lifting pipeline... why not use this, which better adheres to the SA style? and keep naming consistent:
|
yes that would be great! |
|
Note to self, @C-Achard, and @MMathisLab
@MMathisLab, let me know when new version of FMPose3D is released, then I'll proceed with updating the requirements. |
Note: this PR depends on the bleeding-edge code from the FMPose3D repository. After release, the latest FMPose3D dependency needs to be specified in DeepLabCut before merging.
Summary:
This PR adds a dedicated FMPose3D inference path in the PyTorch modelzoo flow and wires early routing for fmpose3d_* models. It relocates FMPose-specific wrappers/inference helpers to pose_estimation_pytorch/modelzoo/fmpose_3d, standardizes outputs to include explicit 3D artifacts (*_3d.h5 / *_3d.json).
Motivation:
Where the first implementation of FMPose3D in #3208 was rather separated from the rest of the modelzoo pipeline, this PR aims to integrate it better in the existing modelzoo video inference pipeline.
Changes:
Revert preliminary implementation in Add fmpose3d functionality #3208:
Route video inference through video_inference_superanimal:
Add brief example usage in the existing superanimal docs
Example usage:
Additional considerations / limitations:
superanimal_name-> means 'dataset family' andmodel_name-> means 'model architecture'. For FMPose3D the 'dataset family' is different and therefore ignored.