Timeline Sequences Data [VE OG]

Data Handling explained how a project is persisted in VidiEditor. It leaves out a detailed explanation of how a timeline with all its different options such as effects, transitions, tracks and other options is modelled and persisted. This article will be focus of this section explaining the different options in detail.

Please note that an object model is implemented that deals with the timeline objects in a flexible and interactive way. This section itself concentrates on how the data looks like that is persisted and can be used to exchange timelines.

Overview of timeline data in VidiEditor

As explained in Data Handling the timeline description (EDL) is stored as metadata on the a projects library. Anyway there are 2 different kind of timeline descriptions used.

• VidiEditor Timeline Data Model

This is a an EDL stored as JSON describing all necessary parameter to visualize the timeline on the users screen graphically and to render the timelines media in the timeline when using web render engine. It is also used in MediaPortal when transforming a VidiEditor timeline to an Adobe Premiere Sequence.

The Model itself will be discussed in deeper detail below.

• VidiCore Sequence Document

This is an EDL usually stored as XML that contains needed information for conforming a timeline using the VidiCoder and controlling the conform output with all the different possibilities VidiVore offers. It is described in detail under: https://apidoc.vidispine.com/5.7/ref/item/sequence.html?highlight=item%20sequence and will not explained here again.

Following figure gives a rough overview what is used where:

Please note that above explanation leaves out VidiFlow. It is not of interest as in case of using VidiFlow workflows for publish also, the flow is the same with the only difference that the Sequence Document is handed over in the workflow first controlling then, the same conforming by a workflow task.

VidiEditor Timeline Data Model (Version 1.0)

Even a small timeline easily results in a JSON that has more then 500 lines. So a complete example will not be presented here. It can be easily generated in VidiEditor while using its log report or EDL export option as described in: Log Report Download and in EDL Import / Export [VE UG].
This section goes through the individual option step by step starting from the highest level and iterating into the document deeper step by step.

Timeline Settings

The top layer describes general settings about the timeline and project used. It has the following elements assuming all sub elements are empty:

{
    "version": "1.0",
    "duration": 15000,
    "id": "ITEM-VX-232937",
    "lastOutTime": 1647,
    "startTimeCode": "00:00:00:00@PAL",
    "title": "timeline",
    "videoTracks": [],
    "audioTracks": [],
    "animationTracks": [],
    "frameRate": {}
}

The elements have following meaning:

• version

The version of the JSON structure used. This might change if this description of the VidiEditor JSON changes to allow proper versioning of the data model for migrations and compatibility questions. New version might be introduced with new releases allowing new options and might require migrations and adoptions.

• duration

The total duration of a timeline in absolute frames. Not meaning its the last frame of media in the timeline but the complete duration of the timeline possible.

• id

The Id of the timeline stored in VidiCore so VidiEditor can reference to the correct item object and persist it.

• lastOutTime

The last visible frame existing on a timeline. This can be different to the total timeline duration presented on the screen.

• startTimeCode

The first frame of the timeline allowing to add a constant offset to the timeline, where the frame count starts at the given value and will count from this values on. It requires the timebase besides the timecode to calculate the absolute frames accordingly. See VidiCore Timebase notation for information about the given notation.

• title

A title for the timeline free to choose by the user in the GUI.

• videoTracks

The video tracks where the video and image segments can be arranged on to create the desired media composition. It holds information about the displayed media and various attributes discussed in th following.

• audioTracks

The audio tracks where the audio segments can be arranged on to create the desired media composition. It holds information about the displayed media and various attributes discussed in th following.

• animationTracks

The 1 animation track where that can be used to add the animated graphics and arrange animated graphics on top of the rest of the timeline.

• frameRate

Stores the frame rate set for the timeline to define the playback timebase as well as the media that can be added to the timeline.

Tracks and Clips

The tracks are an essential element to be able arranging the individual media elements over time and over multiple tracks. There are different track types available called:

• videoTracks

• audioTracks

• animationTracks

But all 3 kinds share the same interface in the object model. So the following parameters are available doesn’t matter for what kind a track is used. Anyway the usage differs depending on the kind of track that will be generated which will be explained for the individual parameters.

A basic example looks as follows:

"videoTracks": [
    {
        "level": 1,
        "mute": false,
        "clips": [
            {
                "clipDuration": 3001,
                "clipSourceIn": 0,
                "id": "ITEM-VX-232143",
                "in": 1153,
                "itemDuration": 3001,
                "itemUniqueId": "59576323-7f5c-d1d4-1d11-027b57c0cc52",
                "linkId": "92b43475-ec56-4509-9e48-a4d4eb021ed2",
                "local": false,
                "mediaType": "Video",
                "name": "",
                "out": 1647,
                "path": "s3://team-vidieditorbucket-s3/voiceover/FILE-VX-14889.mxf?endpoint=s3.eu-central-1.amazonaws.com&region=eu-central-1",
                "sourceIn": 1153,
                "audioMapping": [],
                "effects": [],
                "sourceOut": 1647,
                "isSelected": true,
                "state": "IDLE",
                "trackLevel": 1,
                "trackType": 0,
                "isAudio": false,
                "voiceoverStatus": 1
            }
        ],
        "channelMute": [],
        "effects": [],
        "transitions": [],
        "type": 0,
        "pseudoClips": [],
        "isActive": false,
        "isSelected": true,
        "isLoading": false
    }
]

• level

Defines the ordering of the tracks for display in GUI and Preview using WRE.
Value: 1 based

• mute

Allows to deactivate the track without deleting / adding it. When set to true the playback in WRE will be ignored.

• clips

The individual media segments used on the video track with various parameters possible .

• …Tracks.clips.clipDuration

• …Tracks.clips.clipSourceIn

• …Tracks.clips.id

The ID of the related source media item in VidiCore that is used.

• …Tracks.clips.in

It describes the timecode of the segments in on a timeline, so the position of the first visible frame (video) or first hearable sample (audio) of the medias segment. The value is an absolute frame count.

• …Tracks.clips.itemDuration

The total duration of the source media used on timeline. This is used for example to calculate how much the media can be extended on the timeline (E.g.: on Trimming Segments). The value is an absolute frame count.

• …Tracks.clips.itemUniqueID

A unique ID used to identify each timeline segment individually and explicitly. The value must be a GUID.

• …Tracks.clips.linkID

A unique ID used to identify the possible linking between audio and video segments. The value must be a GUID.

• …Tracks.clips.local

A boolean flag describing if the media is available on a central storage or only on a local PC. This is used for Voice Over recordings but might be used additionally in future for local media files.

• …Tracks.clips.mediaType

The type defines what type of media is used, this can be either video, audio or image depending on the used physical media. Please note that voice Over recordings will be defined as audio while animated graphis are separated via there own track.

• …Tracks.clips.name

A String attribute that can be used for naming the segment, for example VidiEditor uses this to blend in a configurable metadata field to the GUI.

• …Tracks.clips.out

It describes the timecode of the segments out on a timeline, so the position of the last visible frame (video) or last hearable sample (audio) of the medias segment. The value is an absolute frame count.

• …Tracks.clips.path

This holds the path information to a physical file to be referenced for the clip/segment. Depending on the use case this can hold different file path informations. Usually it is used in the context of VidiEditor to reference the file path of the proxy file to be streamed to the client by VidiStream and to be rendered using WRE.

• …Tracks.clips.sourceIn

The sourceIn is used to reference the in point of the source media that is used on the timelines clip in position. So the sourceIn is the absolute frame of the media to be presented at the in position of the segment / clip on timeline.

• …Tracks.clips.audioMapping

A JSON element used for mapping audio channels of clips / segments of mediaType audio. The options will be explained below in an own section.

• …Tracks.clips.effects

A JSON element used for storing informations of effect parameters per clip / segment that will be presented in the playback and GUI. The individual options will be explained in an own section.

• …Tracks.clips.sourceOut

The sourceOut is used to reference the out point of the source media that is used on the timelines clip out position. So the sourceOut is the absolute frame of the media to be presented at the out position of the segment / clip on timeline.

• …Tracks.clips.isSelected

A boolean flag describing if the clip / segment is currently selected or not.

• …Tracks.clips.state

The state usable to define the current state of a clip / segment on timeline. At this time 2 states are defined as clip is either static positioned on timeline (IDLE) or in ove currently by the user (MOVE).

• …Tracks.clips.groupID

A unique identifier as GUID that can bind different clips together. It is used for multiselecting different clips / segments in one go.

• …Tracks.clips.trackLevel
  

Define which track of current clip locate to. Same as Level
Value: 1 based

• …Tracks.clips.trackType

This defines the the kind of track to be of value 0 - video, 1 - audio or 2 - animated graphics.

• …Tracks.clips.isAudio

This boolean flag defines if a track is usable for audio or not. It can be helpful having this explicitly set beside trackType when using the interface.

• …Tracks.clips.voiceoverStatus

A specific state for voice over status determining the status of a voicer over clip / segment. This is used to identify if the clip is local, centrally available or pending. The values are 0 - pending/ingesting, 1 - centrally ingested, 2 - local.

• …Tracks.channelMute

This defines if a dedicated audio channel is muted which differs from the “mute” option used for complete tracks and which is explained above. It can hold to boolean value defining the state per channel.

• …Tracks.effects

This defines effects being applied to a track. In current implementation this is only used for audio leveling on a track while all other effects are handled on clip level. It might be usable when needing more effects are on track level.

• …Tracks.transitions

A JSON element holding the needed informations about transitions used on or in between clips / segments. It will be explained in detail below

• …Tracks.type

• …Tracks.pseudoClips

A specific kind of clips / segments not working as media clips / segments. It is used for gaps between timeline segments being stored under this JSON element.

• …Tracks.isActive

A boolean flag only used for audio tracks holding voice over data. The state describes if the track is currently in recording state or not. If not it acts like a normal audio playback track.

• …Tracks.isSelected

A boolean flag that defines if the track itself is selected or not.

• …Tracks.isLoading

• …Tracks.channelCount

• …Tracks.channelMute

This defines if channels of a track are muted with 2 boolean flags inside an array where the 1st parameter defines for the first channel of the track if it is muted or not and the 2nd one for channel 2.

Tracks.Effects

This can hold effects parameters on a track level. This is similar to effects on a clip / segment level with the difference that only audio leveling is used at this point of time for track effects and keyframes ar not supported on a track effect.

• type = gain

This effect allows to change the level of an audio signal while controlling its gain. It allows to set a gain factor that defines how much the level will be adjusted. A gain of 1 means no adjustment while values between 0 and 1 lower the signal and values above 1 bring up the signal.

{
  "type": "gain",
  "gain": {
      "default": 1.9953
   }
}

Tracks.Clips.Effects

As discussed above video and audio effects can be applied on clip / segment level. This section explains the individual settings possible. In general it can be said that following effects are possible on a clip / segment level:

• Video

  • Opacity

  • Scale

  • Rotation

  • Position

  • Blur

  • Mosaic

• Audio

  • Leveling

An example looks as follows:

"effects": [
                        {
                            "type": "opacity",
                            "opacity": {
                                "default": 0.97
                            }
                        },
                        {
                            "type": "scale",
                            "link": true,
                            "horiz": {
                                "default": 1.04
                            },
                            "vert": {
                                "default": 1.04
                            }
                        },
                        {
                            "type": "rotation",
                            "angle": {
                                "default": 6
                            }
                        },
                        {
                            "type": "position",
                            "y": {
                                "default": 0.07
                            },
                            "x": {
                                "default": -0.08
                            }
                        },
                        {
                            "type": "blur",
                            "id": "252e7408-3b33-4be6-8b30-3d8b624bda34",
                            "blurriness": {
                                "default": 0
                            },
                            "x": {
                                "keyFrames": [
                                    {
                                        "position": 478,
                                        "type": "linear",
                                        "value": 0.28
                                    },
                                    {
                                        "position": 580,
                                        "type": "linear",
                                        "value": 0.71
                                    }
                                ]
                            },
                            "y": {
                                "default": 0.01
                            },
                            "h": {
                                "default": 0.55
                            },
                            "w": {
                                "default": 0.37
                            },
                            "maskType": {
                                "default": 1
                            },
                            "border": {
                                "default": 2
                            }
                        }
                    ]

• type = opacity

Defines the opacity of a clip in percentage from 0 to 100.

{
 "type": "opacity",
 "opacity": {
 "default": 0.97
}

• type = scale

Defines the scaling of of a clip between where 0 is the minimum limit value resulting in a non visible frame while the maximum value can be in theory of any number. Anyway the usage is limited to a maximum border of 200 in VidiEditor for better usage. 100 means a scaling being exactly in the dimensions of the target (E.g. the player dimensions). Note also that horizontal and vertical scaling can be adjusted independently to stretch or squeeze the video.

{
  "type": "scale",
  "link": true,
  "horiz": {
  "default": 1.04
},
 "vert": {
 "default": 1.04
}

• type = rotation

Defines the rotation of the video in degrees. This basically is possible in infinite range of values resulting in repeating rotation. Anyway in VidiEditor the usage is limited in the range from 0 to 360 degrees.

{
"type": "rotation",
"angle": {
"default": 6
 }
}

• type = position

Defines the positioning of of a clip between -1 and +1 where 0 for vertical and horizontal scaling means the video is in center of the player while -1 / +1 means the video is positioned exactly outside of the visible area.

{
"type": "rotation",
"angle": {
"default": 6
 }
}

• type = blur

Adding this effect allows to blur the video picture where 0 means not blurred at all and the max value is set to 100 resulting in a strong blur. Additionally it is possible to set a mask for the blur area to only blur a part of the video. The mask can be either an rectangle or an ellipse. Following parameters exist:

• id - GUI to identify the effect that might be added multiple times

• blurriness 0 -100

• x - the x coordinate / position

• y- the y coordinate / position

• h - the horizontal dimension

• w - the vertical dimension or width

• maskType - Either 0 for rectangle or 1 for ellipse

• border - Integer value defining strength in pixels

{
    "type": "blur",
    "id": "23101375-5d66-48c0-bf43-37aaacdbcef2",
    "blurriness": {
        "default": 46
    },
    "x": {
        "default": 0.3
    },
    "y": {
        "default": 0.02
    },
    "h": {
        "default": 0.58
    },
    "w": {
        "default": 0.35
    },
    "maskType": {
        "default": 1
    },
    "border": {
        "default": 2
    }
}

• type = mosaic

Adding this effect allows to pixelate the video picture in a mosaic kind of style where the horizBlocks and vertBlocks define the strength of the visual effect in number of blocks. Additionally it is possible to set a mask for the mosaic area to only pixelate a part of the video. The mask can be either an rectangle or an ellipse. Following parameters exist:

• id - GUI to identify the effect that might be added multiple times

• horizBlocks 0 -200 (number of blocks to be displayed horizontal)

• vertBlocks 0 -200 (number of blocks to be displayed vertical)

• x - the x coordinate / position

• y- the y coordinate / position

• h - the horizontal dimension

• w - the vertical dimension or width

• maskType - Either 0 for rectangle or 1 for ellipse

• border - Integer value defining strength in pixels

  {
    "type": "mosaic",
    "id": "a680dbfc-6b6a-47b2-9a12-904518a7b933",
    "horizBlocks": {
        "default": 17
    },
    "vertBlocks": {
        "default": 20
    },
    "x": {
        "default": 0
    },
    "y": {
        "default": 0
    },
    "h": {
        "default": 0
    },
    "w": {
        "default": 0
    },
    "maskType": {
        "default": 0
    },
    "border": {
        "default": 2
    }
}

• type = gain

This effect allows to change the level of an audio signal while controlling its gain. It allows to set a gain factor that defines how much the level will be adjusted. A gain of 1 means no adjustment while values between 0 and 1 lower the signal and values above 1 bring up the signal.

{
  "type": "gain",
  "gain": {
      "default": 1.9953
   }
}

Tracks.Clips.Effects.{Parameter}.keyFrames

Keyframes can be used on various effect parameter to adjust the value in a linear way over time where an array of keyframes can be set to vary the effect value. So when 2 keyframes are defined the effect value will change from the earlier to the later keyframes position in a liner way.

• position defines the timecode of the keyframe as absolute frame

• type defines ho the value will be changed over time, only linear is supported at this point of time#

• value defines the acutal value of the keyframe adjusted parameter for this frame position

{
"type": "scale",
"link": true,
"horiz": {
    "keyFrames": [
        {
            "position": 178,
            "type": "linear",
            "value": 0.67
        },
        {
            "position": 276,
            "type": "linear",
            "value": 1.54
        }
    ]
},