Motion Photos are single files that contain a still photographic image and a
short video, which includes audio recording. This type of media allows a user to
view a high resolution still image as well as the video and sound to capture the
sentiment and atmosphere where the image was taken.

## Dependencies

The following are normative references for this specification:

- [Key words for use in RFCs to Indicate Requirement Levels](https://www.ietf.org/rfc/rfc2119.txt)
- [T.81 (09/92) Digital compression and coding of continuous-tone still
  images](http://www.itu.int/rec/T-REC-T.81-199209-I/e) (JPEG)
- [ISO/IEC 23008-12:2022 High efficiency coding and media delivery in
  heterogeneous environments Part 12: Image File Format](https://www.iso.org/standard/83650.html) (HEIC)
- [AV1 Image File Format (AVIF)](https://aomediacodec.github.io/av1-avif/v1.1.0.html) (AVIF)
- [Ultra HDR Image Format v1.0 \| Android Developers](https://developer.android.com/guide/topics/media/platform/hdr-image-format) (Ultra HDR)
- [ISO 16684-1:2011(E) XMP Specification Part 1](https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart1.pdf) (XMP)
- [Adobe XMP Specification Part 3 Storage in Files](https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart3.pdf) (XMP)
- [ISO/IEC 14496-10:2022 Coding of audio-visual objects Part 10: Advanced
  video coding](https://www.iso.org/standard/83529.html) (AVC)
- [ISO/IEC 23008-2:2023 High efficiency coding and media delivery in
  heterogeneous environments Part 2: High efficiency video
  coding](https://www.iso.org/standard/85457.html) (HEVC)
- [AV1 Bitstream \& Decoding Process Specification](https://aomediacodec.github.io/av1-spec/av1-spec.pdf) (AV1)
- [ISO/IEC 14496-1:2010 Coding Of Audio-Visual Objects -
  Systems](https://www.iso.org/standard/55688.html)
- [ISO/IEC 14496-12:2015 ISO Box media file format](https://www.iso.org/standard/68960.html#:%7E:text=ISO%2FIEC%2014496%2D12%3A2015%20specifies%20the%20ISO%20base,such%20as%20audio%2Dvisual%20presentations.) (ISOBMFF)
- [ISO/IEC 14496-14:2020 Coding of audio-visual objects Part 14: MP4 file
  format](https://www.iso.org/standard/79110.html) (MP4)
- [Apple QuickTime File Format](https://developer.apple.com/documentation/quicktime-file-format) (MOV)

## Introduction

The use of "MUST", "MUST NOT", "REQUIRED", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" is per the IETF standard
defined in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).

## Motion Photo Format

Motion Photo files consist of a primary still image file,
[JPEG](https://www.itu.int/rec/T-REC-T.81-199209-I/e), [HEIC](https://www.iso.org/standard/69668.html), or [AVIF](https://aomediacodec.github.io/av1-avif/v1.1.0.html), with a
secondary video file appended to it. The primary image contains Camera XMP
metadata describing how to display the still image file and video file contents,
and Container XMP metadata describing how to locate the video file contents.

The image file may have a gainmap, as is the case with [Ultra HDR JPEGs](https://developer.android.com/guide/topics/media/platform/hdr-image-format).

### Filename Pattern

Writers should use a filename matching the following regular expression:  

    ^([^\\s\\/\\\\][^\\/\\\\]*MP)\\.(JPG|jpg|JPEG|jpeg|HEIC|heic|AVIF|avif)

Readers may ignore the XMP metadata, the appended video file, or the video
contents if the pattern is not followed.

### Media Data Encoding

The primary image contains a Container Element [XMP](https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart1.pdf)
metadata directory defining the order and properties of the subsequent media
file in the file container. Each file in the container has a corresponding media
item in the directory. The media item describes the location in the file
container and the basic properties of each concatenated file.

### XMP Attributes

Two sets of XMP metadata are used to define the extra semantic information for
the Motion Photo format. The metadata may appear in any order.

#### Camera Metadata

Camera metadata encodes information about how to present the primary image and
video portions of the Motion Photo.

- The namespace URI is `http://ns.google.com/photos/1.0/camera/`
- The default namespace prefix is `Camera`

The following attributes may appear in the still image file XMP metadata:

|---|---|---|
| Name | Type | Description |
| `Camera:MicroVideo` `Camera:MicroVideoVersion` `Camera:MicroVideoOffset` `Camera:MicroVideoPresentation` ` TimestampUs` |   | These properties were part of the Microvideo V1 specification. They are deleted in this specification and must be ignored if present. In particular, the `MicroVideoOffset` attribute is replaced by the `GContainer:ItemLength` value for locating the video data in the file. |
| `Camera:MotionPhoto` | `Integer` | 0: Indicates that the file shouldn't be treated as a Motion Photo. 1: Indicates that the file should be treated as a Motion Photo. All other values are undefined and are treated equivalently to 0. If the value is zero or negative, then the file is always treated as a non-Motion Photo, even if a video is in fact appended to the file. Since XMP is carried over by most well behaved editors, still image files may still have a residual value of 1 for this field even though the appended video has been stripped. This field is therefore not definitive and readers must always confirm that a video is present. |
| `Camera:MotionPhotoVersion` | `Integer` | Indicates the file format version of the Motion Photo. This specification defines version "1". |
| `Camera:MotionPhotoPresentationTimestampUs` | `Long` | Long value representing the presentation timestamp (in microseconds) of the video frame corresponding to the image still. Value can be -1 to denote unset/unspecified. |

##### Presentation behavior

If `Camera:MotionPhotoPresentationTimestampUs` is not present in the XMP packet,
readers should use a presentation timestamp immediately preceding the timestamp
closest to the middle of the video track, i.e., the duration of the video track
divided by 2.

If `Camera:MotionPhotoPresentationTimestampUs` is present in the XMP packet and
the `"application/motionphoto-image-meta"` is present in the video, the same
value must appear in the `primaryImageFrameScoreDescr` `presentationTimestampUs`
field of that track. If `Camera:MotionPhotoPresentationTimestampUs` is not
present in the XMP packet and the metadata track is present, then the value in
the metadata track must be -1.

#### Container element

The container element is encoded into the XMP metadata of the primary image and
defines the directory of media items in the container. Media items must be
located in the container file in the same order as the media item elements in
the directory and must be tightly packed.

- The namespace URI is `http://ns.google.com/photos/1.0/container/`
- The default namespace prefix is `Container`

The directory may contain only one primary image item and it must be the first
item in the directory.

|---|---|---|
| Element Name | Type | Description |
| `Directory` | Ordered Array of Structures | Ordered array of `Container:Item` structures defining the layout and contents of the container. |

#### Item element

Media item elements describe how each item should be used by the application.

- The namespace URI is `http://ns.google.com/photos/1.0/container/item/`
- The default namespace prefix is `Item`

The first media item must be the primary image. It must contain a `Mime`
attribute specifying one of the image MIME types listed in Item MIME Type
Values. The length of the primary item may be determined by parsing the primary
image based on its MIME type starting at the beginning of the file container.

The first media item may contain a `Padding` attribute specifying additional
padding between the end of the encoded primary image and the beginning of the
next media item. Only the first media item may contain a `Padding` attribute.

Each media item must contain a `Mime` attribute. The secondary media items must
also contain Length attributes.

Sequential media items may share resource data within the file container. The
first media item determines the location of the resource in the file container,
and subsequent shared media items have `Length` set to 0 in the case that the
resource data is itself a container.

The location of media item resources in the container is determined by summing
the `Length` values of the preceding secondary item resources to the length of
the primary image encoding plus `Padding` if specified.

|---|---|---|
| Attribute Name | Type | Description |
| `Mime` | `String` | Required. Simple string indicating the MIME type of the media item in the container. |
| `Semantic` | `String` | Required. Simple string indicating the application specific meaning of the media item. See Item Semantic Values section for definition. |
| `Length` | `Integer` | Required for secondary media items, including the video container. The positive integer length in bytes of the item. Media items are expected to be in their original form, with no encoding applied. The length value is the actual length of the bytes in the file. Length 0 in secondary media items indicates that the media item resource is shared with the previous media item. Length is expected to be 0 in a primary media item. |
| `Padding` | `Integer` | \[JPEG-based motion photos\] Optional for the primary media item. Simple string containing a positive integer length in bytes of additional padding between the end of encoded primary image and the beginning of the next media item. \[HEIC/AVIF-based motion photos\] Required for the primary media item. Must have a value equal to 8, the header length of [the Motion Photo Video Data ("mpvd") box](https://developer.android.com/media/platform/motion-photo-format#isobmff-image-specific-behavior). |

| **Non-normative:** The `Length` attribute for the secondary media item enables readers to quickly locate the video without reading the entire container file. Readers can start from the end of the file and move back `<Length>` bytes to find the start of the video.
|
| Writers don't have to determine the length of the primary still image file when writing the XMP within it since Length is an optional attribute for the primary item.

##### Item:Mime type values

The `Item:Mime` attribute defines the MIME type of each media item.

|---|---|
| Value | Description |
| `image/jpeg` | [JPEG](http://www.itu.int/rec/T-REC-T.81-199209-I/e) image |
| `image/heic` | [HEIC](http://ISO/IEC 23008-2:2017 High efficiency video coding) image |
| `image/avif` | [AVIF](https://aomediacodec.github.io/av1-avif/v1.1.0.html) image |
| `video/mp4` | [MP4](https://www.iso.org/standard/79110.html) container |
| `video/quicktime` | [MOV](https://developer.apple.com/documentation/quicktime-file-format) container |

##### Item:Semantic values

The `Item:Semantic` attribute defines the application specific meaning of each
media item in the container directory.

|---|---|
| `Value` | Description |
| `Primary` | Indicates that the media item is the primary display ready image in the container. A motion photo must include one, and only one, item with this semantic. |
| `MotionPhoto` | Indicates that the media item is the video container. A motion photo must include one, and only one, item with this semantic. The location of this media item must be at the end of the file. No other bytes may be placed after this media item's bytes have terminated. |

###### Motion photos with a primary Ultra HDR image

As per the item semantic value rules defined in this specification and in
[the Ultra HDR image specification](https://developer.android.com/guide/topics/media/platform/hdr-image-format#GContainer_details),
motion photos with a primary Ultra HDR image must also include a media item with
an item semantic of `"GainMap"`. Also, writers encoding motion photos must place
the gainmap item element before the video item element.

### ISOBMFF-image-specific behavior

Motion Photos with
[ISOBMFF-based](https://www.iso.org/standard/68960.html#:%7E:text=ISO%2FIEC%2014496%2D12%3A2015%20specifies%20the%20ISO%20base,such%20as%20audio%2Dvisual%20presentations.)
images (e.g., HEIC and AVIF images) must have a structure such that the image
portion of the file terminates with a top-level "Motion Photo Video Data" box,
described using the semantics of the Syntactic Description Language defined in
ISO/IEC 14496-1:2010(E) clause 8:  

    // Box as defined in ISO/IEC 14496-12:2015: 4.2
    aligned(8) class MotionPhotoVideoData extends Box('mpvd') {
      bit(8) data[];
    }

where the "data" field contains all of the video bytes. The special value of "0"
is not allowed for the size of the Motion Photo Video Data Box. (Refer to
[ISO/IEC 14496-12:2015: 4.2](https://www.iso.org/standard/55688.html) for the definition of a size of class that
extends Box.)

The ISOBMFF image's XMP must also define the primary media item's Padding
attribute value as being equal to the size in bytes of the Motion Photo Video
Data Box header, that is the size and name headers.

Refer to Figure 1 illustrating this box structure for a sample HEIC-based motion
photo:

![Line diagram demonstrating the arrangement of elements in an HEIC motion file](https://developer.android.com/static/images/media/platform/motion-photo-heic.png)

**Figure 1.** Illustration of the top-level boxes of a sample HEIC image within
a single HEIC motion photo file. Note that the order of boxes is mostly for
illustration only (please refer to the relevant standards on how to construct
HEIF or video files); nevertheless, the "mpvd" box must come after all the HEIC
image file's boxes.

### Video container contents

The video container file that is appended to the primary image must include at
least one primary video track. This track is required and contains video encoded
in [AVC](https://www.iso.org/standard/83529.html),
[HEVC](https://www.iso.org/standard/85457.html), or
[AV1](https://aomediacodec.github.io/av1-spec/av1-spec.pdf). The
primary video frame resolution is undefined. The video color space, transfer
function, and bit-depth may vary. For example, SDR videos may have an 8-bit bit
depth, BT.709 color space, with an sRGB transfer function. Or, HDR videos may
have a 10-bit bit depth, BT.2100 color space, and an HLG or PQ transfer
function, along with HDR metadata and metadata tracks.

The video container file may contain one optional higher resolution secondary
video track. Readers should use its contents to display substitutions for the
primary still image encoded in the JPEG or HEIC image. This track may contain
lower frame rate video encoded in AVC, HEVC, or AV1. The secondary video frame
resolution is undefined.

It is expected that all of the frames in the secondary video track have
corresponding frames in the primary video track. Each pair of corresponding
frames in primary and secondary video tracks should have identical presentation
timestamps. If there is a secondary track frame without corresponding primary
track frame, viewers can try selecting a primary track frame with the closest
matching presentation timestamp as the representative thumbnail for that
secondary video track.

The video container file may contain one optional 16-bit mono or stereo audio
track at 44kHz, 48 kHz, or 96 kHz encoded in AAC. Readers should present this
audio track when the primary video track is displayed.

The secondary video track, if present, should always come *after* the primary
video track. There are no other ordering constraints regarding other tracks. The
primary video track must have a track index lower than that of any secondary
video track. That is, if the primary video track has a track number 2, then any
secondary video track must have a track number greater than or equal to 3.

### Video Metadata Track with Machine Intelligence Scoring

Writers may optionally add a metadata track to the video container file with a
type "meta". The metadata track should have exactly one sample that contains a
byte stream in [the format described in "Syntax"](https://developer.android.com/media/platform/motion-photo-format#metadata-syntax).

If the metadata track is present, the sample description table entry for the
track (i.e., the "stsd" box located at "mdia.minf.stbl.stsd" relative to the
"trak" box) must contain a single atom that indicates a text metadata sample
entry - (i.e, a "mett" box). The "mett" box must have a MIME type string equal
to "application/motionphoto-image-meta".

#### Syntax

If this metadata track is defined, its content must consist of a byte stream
conforming to this `MotionPhotoMetadataDescriptor` specification, described
here using the semantics of the Syntactic Description Language defined in
ISO/IEC 14496-1:2010(E) clause 8.  

    // BaseDescriptor as defined in ISO/IEC 14496-1:2010(E): 7.2.2.2
    abstract aligned(8) expandable((1<<28) - 1) class BaseDescriptor
        : bit(8) tag=0 {
      // Empty. To be filled by classes extending this class.
    }

    // Score data for a frame.
    class MotionPhotoFrameScoreDescriptor extends BaseDescriptor
        : bit(8) tag=MotionPhotoFrameScoreDescrTag {
      // The frame's score in the range [0, 1].
      float(32) score;

      // The frame's presentation timestamp in microseconds.
      int(64) presentationTimestampUs;
    }

    // Score data for a track.
    class MotionPhotoTrackScoreDescriptor extends BaseDescriptor
        : bit(8) tag=MotionPhotoTrackScoreDescrTag {
      // The number of scored frames in the track.
      unsigned int(32) numScoredFrames;

      // The track's frames' score data. They must be in ascending order with
      // respect to the presentation timestamp.
      MotionPhotoFrameScoreDescriptor trackFrameScoreDescr[numScoredHighResFrames];
    }

    // Score data for a motion photo.
    class MotionPhotoScoreDescriptor extends BaseDescriptor
        : bit(8) tag=MotionPhotoScoreDescrTag {

      // Machine-intelligence model version used to calculate the scores. Writers
      // using a scoring model should set this field to 1 or greater. Writers not
      // using any scoring model should set this field to 0.
      unsigned int(32) modelVersion;

      // The primary image's frame score data.
      MotionPhotoFrameScoreDescriptor primaryImageFrameScoreDescr;

      // The high-resolution motion photo frames' score data.
      MotionPhotoTrackScoreDescriptor highResTrackScoreDescr;
    }

    // Flag data for a track.
    class MotionPhotoTrackFlagsDescriptor extends BaseDescriptor
        : bit(8) tag=MotionPhotoTrackFlagDescrTag {
      // Set to true to indicate the video frames have been stabilized and don't
      // require readers of the track to apply any further stabilization.
      bit(1) isStabilized;
    }

    // Flags for a motion photo.
    class MotionPhotoFlagsDescriptor extends BaseDescriptor
            : bit(8) tag=MotionPhotoFlagDescrTag {
      // The low-resolution motion photo track's flag data.
      MotionPhotoTrackFlagDescriptor lowResTrackFlagsDescr;

      // The high-resolution motion photo track's flag data.
      MotionPhotoTrackFlagDescriptor highResTrackFlagsDescr;
    }

    // Container for motion photo metadata, like stabilization indicators and
    // quality scoring.
    class MotionPhotoMetadataDescriptor extends BaseDescriptor
        : bit(8) tag=MotionPhotoMetadataDescrTag {
      // Scoring data for the still and high-res frames.
      MotionPhotoScoreDescriptor motionPhotoScoreDescr;

      // Flags for the low-res and high-res frames.
      MotionPhotoFlagDescriptor motionPhotoFlagDescr;
    }

    // Class tags for MotionPhotoData using the "User Private" tag space 0xC0-0xFE
    // for descriptors defined in ISO/IEC 14496-1:2010(E): 7.2.2.1, Table 1.
    // 0xC0 MotionPhotoMetadataDescrTag
    // 0xC1 MotionPhotoScoreDescrTag
    // 0xC2 MotionPhotoTrackScoreDescrTag
    // 0xC3 MotionPhotoFrameScoreDescrTag
    // 0xC4 MotionPhotoFlagsDescrTag
    // 0xC5 MotionPhotoTrackFlagDescrTag