X Tutup

Boximator:

Generating Rich and Controllable Motions for Video Synthesis

Jiawei Wang*,
Yuchen Zhang*,
Jiaxin Zou,
Yan Zeng,
Guoqiang Wei,
Liping Yuan,
Hang Li
ByteDance Research   *Equal Contribution
arXiv Video Demo Cite


Examples

Base Model: PixelDance
"The kitten is hiding herself into the cup" "An orange kitten is poking its head out of the tea cup." "A man is dancing before the background full of colorful pixels." "The character made of pixels is dancing."
"The female warrior lefts up her sword." "The female warrior switches her sword from left hand to right hand." "A boy and a girl are kissing." "A boy and a girl are kissing."
"A girl in red is covering her face with a skull." "Skull in the watercolor style. Butterflies are flying around." "A whale jumps out of water." "A whale falls back to water."
"A cat is chasing a ball." "A dog is chasing a red ball." "Drone flying over New Zealand beach." "Drone turning around over New Zealand beach to face the ocean."
"The knight rides down to the wild." "The knight rides down to the wild, camera following his back." "Anime girl playing piano." "Two animation roles sitting on the moon towards the earth, they are getting closer and closer."
"A bird is jumping left and right on the branch." "A bird with grey, red and yellow feathers flys away." "A young woman is turning her head, revealing her face in profile." "A girl is covering her face with hands."
"Spiderman swings towards the camera." "A cute fluffy hamster pilot walking towards a fighter aircraft." "Fat black man fighting a polar bear and run away, anime style." "A wolf howled at the moon and then jumped down from the stone and ran away."
"A cat is jumping to a table." "A woman is running on the street with a dog." "Hot air balloons floating in Turkey sky." "The gril flys her red balloon away under the sunset."
"A man sitting on a table is drinking a cup of coffee." "An astronaut is skateboarding on the moon." "Two boxers are fighting" "The flower swings in the wind."

Comparisons with Other Methods

Note: Pika and Gen-2 use image and text conditions; Boximator uses additional box constraints derived from the text prompt.
Boximator Pika 1.0 Gen-2

"A cute 3D boy is standing and then walking."

"The adorable pika turns to the camera."

"Very close view of a beautiful girl closing up her eyes."

"A handsome man is taking out a rose from his pocket with his right hand and looking at the rose."

"The wind blows a woman's umbrella away, rainy day."

"Two raccoons in blue shirts are playing a ball, the left one is jumping up."

"Camera rotate in a bedroom, showing a big landscape painting on the wall."

"Adding wine to a glass."

Try It Yourself

Our demo website is under development and will be available in the next 2-3 months. We will attach the demo link on this website once the demo is ready. No extra application process is required. Before that, we have prepared an early experience channel for everyone: You can try Boximator by emailing wangjiawei.424@bytedance.com. We will reply to you with the generated video as soon as possible.

Users' email should include:
  • An image as the initial frame
  • A text prompt to describe the motion you need

Tips:
  • Please send us a clean original image. The aspect ratio of the image is preferably 1:1.
  • You are encouraged to use detailed words to describe your needs, especially about the motion. Don't worry about being lengthy. ^_^
  • By default, user-provided data along with the according generated video are allowed to be shown in our gallery. Otherwise, please indicate your requirements in the email.
  • Please do not send any images or texts that may be pornographic, political, bloody, etc.
UPDATE
  • 2024/02/18: Hello everyone! We are back to work! Due to the previous Lunar New Year vacation, we have accumulated some user emails and samples, so it may take you some time to wait our reply. User samples suitable for Boximator will be prioritized. Thanks for your patience!

  • 2024/02/07: Happy Lunar New Year! We will be on vocation during 2024.02.08~02.17. All emails sent before February 7th have been responded. Thanks for your attention!

Look forward to Boximator colliding with your imagination to create unique sparks!
Don't hesitate to contact us.

BibTeX

@misc{wang2024boximator,
      title={Boximator: Generating Rich and Controllable Motions for Video Synthesis}, 
      author={Jiawei Wang and Yuchen Zhang and Jiaxin Zou and Yan Zeng and Guoqiang Wei and Liping Yuan and Hang Li},
      year={2024},
      eprint={2402.01566},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
X Tutup