Storage Configuration & Access [C ARC]

Motivation

Concepts for integrating storages on the Kubernetes cluster level are described in the Storage Concepts [C ARC] section. When it comes to using these storages in an Enterprise MAM Solution they need to be configured properly to be accessible for all components of the system.

Concepts

VidiCore distinguishes between the abstract notion of a storage location and different ways of physical access to these storages.

The abstract notion of a a storage is described by a VidiCore storage. For accessing the physical files on a storage you can configure one or more storage methods in VidiCore. They define how the storage can be reached. A storage may have several storage methods if the storage can be reached via different protocols.

Storages can be grouped into storage groups for specific purposes, e.g. quota management.

Please note that some physical locations for storing files are modelled by further concepts in VidiCore. This applies to “fire & forget” storages as well as to thumbnails. See below for more details.

Managed And Unmanaged Storages

A storage in VidiCore always contains files that are references by a VidiCore entity (at least a file entity, mostly also an item). So-called “fire & forget” storages where VidiCore does not keep track of the files in them are modelled as export locations.

How To Access A Storage

A storage method is an URI describing how to physically access a storage. A storage can have different methods but all methods must point to the same physical location. VidiCore regularly scans all storage methods by default. If the storage can be reached it will get the state READY otherwise it will have state OFFLINE. Each storage method keeps track of when the last successful access has happened and – if applicable – the last failure.

The URI schemes supported by VidiCore are documented here in VidiCore APIdoc.

Additional notes:

Storage method URIs inside a StorageDocument need to be URI-escaped to be able to transport special characters like \ or %. So the proper representation of an ftp account with username=DOMAIN\USERpassword=Hallo%XX would be: <uri>ftp://DOMAIN%5CUSER:Hallo%25XX@server.company.com/</uri>

A storage method can contain any well-formed URI. By this, VidiFlow can store access methods on a VidiCore storage that VidiCore itself cannot access. See below for more details.

VidiCore only uses a single storage method for monitoring the storage and uses this logic to select the storage method for this purpose:

1. check the connectivity for all the methods regardless of their browse flag value;

2. pick the most effective method to use for scanning, with these conditions (all must meet):

   1. the method succeeds to connect/access by VidiCore/VSA;

   2. the browse flag is true;

   3. the method’s type is NONE (or empty when creating the method).

You can prevent VidiCore from using a method for scanning the storages by setting the browse flag to false on this method.

Storages For Different Asset Types (Proxy, Hires, Thumbnails)

By default all VidiCore storages are equivalent. In particular, no distinction is made between hires and proxy storages. If the systems needs to store a file, it will use the LOCAL storage that is online with highest remaining capacity (see http://apidoc.vidispine.com/latest/storage/storage.html#items-and-storages).

To force VidiCore to use a certain storage, these methods are available:

• A default ingest storage can be defined.

• For import operations a storage can explicitly be selected.

• For transcode operations a target storage can be specified.

• Storage rules can be used to place certain shape tags to certain storages, see http://apidoc.vidispine.com/latest/storage/storage-rule.html and http://apidoc.vidispine.com/latest/ref/storage/storage-rule.html#creating-modifying-reading-storage-rules.

• Storage cleanup can be disabled for preventing VidiCore moving files away from a storage (e.g. for browse files that always should reside on a specific storage) - see http://apidoc.vidispine.com/latest/storage/storage.html#storage-cleanup

An Enterprise MAM solution follows the idea of making storage selection explicit. Thus, it is up to a VidiFlow workflow to decide for source and destination storages and pass this information to all Agents [C ARC] doing file-based operations-

The following table gives an overview of how different types of storage locations are modelled in VidiCore.

Storage Methods Not For VidiCore

For some scenarios storage methods need to be defined that are not intended to be used by VidiCore but for other modules of an Enterprise MAM Solution. For such methods set a storage method different to NONE to prevent VidiCore from using it.

Such a storages method may be an additional storage method on top of the ones usable by VidiCore to model alternative access methods to the storage (which may be faster, more efficient, etc.).

There may also be storages which VidiCore cannot access at all (e.g. some video servers) and the non-VidiCore storage method is the only one available. In this case it has to be ensured that at all operations on this storage (e.g. housekeeping, copying files from and to the storage) are implemented by VidiFlow-based agents.

For deleting files it has to be ensured that not only the physical file but also the VidiCore FILE entity is properly deleted:

1. delete physical file with the storage’s / video server’s API;

2. unregister VidiCore FILE entity for that file;

3. if required, delete the VidiCore item.

The following method types are pre-defined an can be used in ConfigPortal:

Use-case	Storage Method Type in VidiCore	Allowed URI Schemes
Access to storage via VidiCore	NONE	all VidiCore-supported schemes
Access to storage via UNC paths by Windows backend modules	WINDOWS	UNC paths as unc://server/share/subdir . Credentials need to be stored separately as you cannot read out a password stored in VidiCore.
Access to storage via mount points paths by Mac compontents	MAC	file:///Volumes/mountpoint (same structure as VidiCore file URIs)
Access to proxy storage by VidiStream	PROXYSTREAMING	Any scheme accessible by VidiStream The storage method URI in VidiCore must not contain credentials.
http-based up- and download by EditMate (e.g. media files, project files) or other products (but not VidiCore)	HTTP	http or https
Client-based access to a storage by EditMate: the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation	EDITMATECLIENT	file
Client-based access to a storage by Adobe Premiere: used for FCP XML export (e.g. by VidiEditor), the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation	FINALCUTXMLEXPORT	file for network paths or local mounts.
For referencing clips on Grass Valley Momentum	GVMOMENTUM	momentum://… (to be defined)
Addressing a file via StorNext API, e.g. for moving a file from disk to tape.	HSM (already defined in VidiCore)	stornext://… (as defined by VidiCore)