Motivation

Concepts for integrating storages on the Kubernetes cluster level are described in the Storage Concepts [GL ARC] section. When it comes to using these storages in an Enterprise MAM Solution they need to be configured properly to be accessible for all components of the system.

Concepts

VidiCore distinguishes between the abstract notion of a storage location and different ways of physical access to these storages.

The abstract notion of a a storage is described by a VidiCore storage. For accessing the physical files on a storage you can configure one or more storage methods in VidiCore. They define how the storage can be reached. A storage may have several storage methods if the storage can be reached via different protocols.

Storages can be grouped into storage groups for specific purposes, e.g. quota management.

Please note that some physical locations for storing files are modelled by further concepts in VidiCore. This applies to “fire & forget” storages as well as to thumbnails. See below for more details.

Managed And Unmanaged Storages

A storage in VidiCore always contains files that are references by a VidiCore entity (at least a file entity, mostly also an item). So-called “fire & forget” storages where VidiCore does not keep track of the files in them are modelled as export locations.

How To Access A Storage

A storage method is an URI describing how to physically access a storage. A storage can have different methods but all methods must point to the same physical location. VidiCore regularly scans all storage methods by default. If the storage can be reached it will get the state READY otherwise it will have state OFFLINE. Each storage method keeps track of when the last successful access has happened and – if applicable – the last failure.

The URI schemes supported by VidiCore are documented here in VidiCore APIdoc.

Additional notes:

URI scheme

Remarks

file

Unix path pointing to locally mounted file systems (i.e. no UNC paths)

smb, smb2

SMB shares in the network, e.g. smb2://user:password@server/share/.

Caution: As VidiCore does not support all SMB protocol versions and features, the usage of smb and smb2 storage methods in an Enterprise MAM solution is strongly discouraged. For making SMB storages available to VidiCore, mount them to /vpmsmounts (see Storage Concepts [GL ARC]) and use file URIs for VidiCore storages methods.

ftp

Active/passive FTP transfer is configured via the URI parameter passive=false (http://apidoc.vidispine.com/latest/storage/storage.html#ftp)

For server-to-server FTP (FXP) both storage methods need to have the metadata entry fxp=true set (key=fxp, value=true).  If it is set on the storage methods of both storages, VidiCore will automatically do FXP when transferring file between these two storages.

Storage method URIs inside a StorageDocument need to be URI-escaped to be able to transport special characters like \ or %. So the proper representation of an ftp account with username=DOMAIN\USERpassword=Hallo%XX would be: <uri>ftp://DOMAIN%5CUSER:Hallo%25XX@server.company.com/</uri>

A storage method can contain any well-formed URI. By this, VidiFlow can store access methods on a VidiCore storage that VidiCore itself cannot access. See below for more details.

VidiCore only uses a single storage method for monitoring the storage and uses this logic to select the storage method for this purpose:

  1. check the connectivity for all the methods regardless of their browse flag value;

  2. pick the most effective method to use for scanning, with these conditions (all must meet):

    1. the method succeeds to connect/access by VidiCore/VSA;

    2. the browse flag is true;

    3. the method’s type is NONE (or empty when creating the method).

You can prevent VidiCore from using a method for scanning the storages by setting the browse flag to false on this method.

Storages For Different Asset Types (Proxy, Hires, Thumbnails)

By default all VidiCore storages are equivalent. In particular, no distinction is made between hires and proxy storages. If the systems needs to store a file, it will use the LOCAL storage that is online with highest remaining capacity (see http://apidoc.vidispine.com/latest/storage/storage.html#items-and-storages).

To force VidiCore to use a certain storage, these methods are available:

An Enterprise MAM solution follows the idea of making storage selection explicit. Thus, it is up to a VidiFlow workflow to decide for source and destination storages and pass this information to all Agents [GL ARC] doing file-based operations-

The following table gives an overview of how different types of storage locations are modelled in VidiCore.

Type of stored files

VidiCore solution

thumbnails (keyframes)

thumnail resource http://apidoc.vidispine.com/latest/system/thumbnails.html

proxy video

LOCAL storage http://apidoc.vidispine.com/latest/storage/storage.html.

Proxy storage(s) are not specially marked in VidiCore, only in VidiFlow’s configuration. VidiFlow workflows need to ensure that browse videos are only put to a browse storage by the transcoders. Additionally, it has to be ensured that VidiCore does not relocate browse videos during storage clean-up.

hires video (central storage), referenced by VidiFlow

LOCAL storage http://apidoc.vidispine.com/latest/storage/storage.html 

hires video (other storages like playout server), referenced by VidiFlow

LOCAL storage http://apidoc.vidispine.com/latest/storage/storage.html 

hires video not referenced by VidiFlow (“fire & forget” storages)

export location
http://apidoc.vidispine.com/latest/ref/export-location.html

 

Storage Methods Not For VidiCore

For some scenarios storage methods need to be defined that are not intended to be used by VidiCore but for other modules of an Enterprise MAM Solution. For such methods set a storage method different to NONE to prevent VidiCore from using it.

Such a storages method may be an additional storage method on top of the ones usable by VidiCore to model alternative access methods to the storage (which may be faster, more efficient, etc.).

There may also be storages which VidiCore cannot access at all (e.g. some video servers) and the non-VidiCore storage method is the only one available. In this case it has to be ensured that at all operations on this storage (e.g. housekeeping, copying files from and to the storage) are implemented by VidiFlow-based agents.

For deleting files it has to be ensured that not only the physical file but also the VidiCore FILE entity is properly deleted:

  1. delete physical file with the storage’s / video server’s API;

  2. unregister VidiCore FILE entity for that file;

  3. if required, delete the VidiCore item.

The following method types are pre-defined an can be used in ConfigPortal:

Use-case

Storage Method Type in VidiCore

Allowed URI Schemes

Access to storage via VidiCore

NONE

all VidiCore-supported schemes

Access to storage via UNC paths by Windows backend modules

WINDOWS

UNC paths as unc://server/share/subdir . Credentials need to be stored separately as you cannot read out a password stored in VidiCore.

Access to storage via mount points paths by Mac compontents

MAC

file:///Volumes/mountpoint (same structure as VidiCore file URIs)

Access to proxy storage by VidiStream

PROXYSTREAMING

Any scheme accessible by VidiStream

The storage method URI in VidiCore must not contain credentials.

http-based up- and download by EditMate (e.g. media files, project files) or other products (but not VidiCore)

HTTP

http or https

Client-based access to a storage by EditMate: the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation

EDITMATECLIENT

file

Client-based access to a storage by Adobe Premiere: used for FCP XML export (e.g. by VidiEditor), the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation

FINALCUTXMLEXPORT

file for network paths or local mounts.

For referencing clips on Grass Valley Momentum

GVMOMENTUM

momentum://… (to be defined)

Addressing a file via StorNext API, e.g. for moving a file from disk to tape.

HSM (already defined in VidiCore)

stornext://… (as defined by VidiCore)