Storage Configuration & Access [C ARC]
Motivation
Concepts for integrating storages on the Kubernetes cluster level are described in the Storage Concepts [C ARC] section. When it comes to using these storages in an Enterprise MAM Solution they need to be configured properly to be accessible for all components of the system.
Concepts
VidiCore distinguishes between the abstract notion of a storage location and different ways of physical access to these storages.
The abstract notion of a a storage is described by a VidiCore storage. For accessing the physical files on a storage you can configure one or more storage methods in VidiCore. They define how the storage can be reached. A storage may have several storage methods if the storage can be reached via different protocols.
Storages can be grouped into storage groups for specific purposes, e.g. quota management.
Please note that some physical locations for storing files are modelled by further concepts in VidiCore. This applies to “fire & forget” storages as well as to thumbnails. See below for more details.
Managed And Unmanaged Storages
A storage in VidiCore always contains files that are references by a VidiCore entity (at least a file entity, mostly also an item). So-called “fire & forget” storages where VidiCore does not keep track of the files in them are modelled as export locations.
How To Access A Storage
A storage method is an URI describing how to physically access a storage. A storage can have different methods but all methods must point to the same physical location. VidiCore regularly scans all storage methods by default. If the storage can be reached it will get the state READY
otherwise it will have state OFFLINE
. Each storage method keeps track of when the last successful access has happened and – if applicable – the last failure.
The URI schemes supported by VidiCore are documented here in VidiCore APIdoc.
Additional notes:
URI scheme | Remarks |
---|---|
| Unix path pointing to locally mounted file systems (i.e. no UNC paths) |
| SMB shares in the network, e.g. Caution: As VidiCore does not support all SMB protocol versions and features, the usage of |
| Active/passive FTP transfer is configured via the URI parameter For server-to-server FTP (FXP) both storage methods need to have the metadata entry |
Storage method URIs inside a StorageDocument
need to be URI-escaped to be able to transport special characters like \
or %
. So the proper representation of an ftp account with username=DOMAIN\USER
password=Hallo%XX
would be: <uri>ftp://DOMAIN%5CUSER:Hallo%25XX@server.company.com/</uri>
A storage method can contain any well-formed URI. By this, VidiFlow can store access methods on a VidiCore storage that VidiCore itself cannot access. See below for more details.
VidiCore only uses a single storage method for monitoring the storage and uses this logic to select the storage method for this purpose:
check the connectivity for all the methods regardless of their
browse
flag value;pick the most effective method to use for scanning, with these conditions (all must meet):
the method succeeds to connect/access by VidiCore/VSA;
the
browse
flag istrue
;the method’s type is
NONE
(or empty when creating the method).
You can prevent VidiCore from using a method for scanning the storages by setting the browse
flag to false
on this method.
Storages For Different Asset Types (Proxy, Hires, Thumbnails)
By default all VidiCore storages are equivalent. In particular, no distinction is made between hires and proxy storages. If the systems needs to store a file, it will use the LOCAL
storage that is online with highest remaining capacity (see http://apidoc.vidispine.com/latest/storage/storage.html#items-and-storages).
To force VidiCore to use a certain storage, these methods are available:
A default ingest storage can be defined.
For import operations a storage can explicitly be selected.
For transcode operations a target storage can be specified.
Storage rules can be used to place certain shape tags to certain storages, see http://apidoc.vidispine.com/latest/storage/storage-rule.html and http://apidoc.vidispine.com/latest/ref/storage/storage-rule.html#creating-modifying-reading-storage-rules.
Storage cleanup can be disabled for preventing VidiCore moving files away from a storage (e.g. for browse files that always should reside on a specific storage) - see http://apidoc.vidispine.com/latest/storage/storage.html#storage-cleanup
An Enterprise MAM solution follows the idea of making storage selection explicit. Thus, it is up to a VidiFlow workflow to decide for source and destination storages and pass this information to all Agents [C ARC] doing file-based operations-
The following table gives an overview of how different types of storage locations are modelled in VidiCore.
Type of stored files | VidiCore solution |
---|---|
thumbnails (keyframes) | thumnail resource http://apidoc.vidispine.com/latest/system/thumbnails.html |
proxy video |
Proxy storage(s) are not specially marked in VidiCore, only in VidiFlow’s configuration. VidiFlow workflows need to ensure that browse videos are only put to a browse storage by the transcoders. Additionally, it has to be ensured that VidiCore does not relocate browse videos during storage clean-up. |
hires video (central storage), referenced by VidiFlow |
|
hires video (other storages like playout server), referenced by VidiFlow |
|
hires video not referenced by VidiFlow (“fire & forget” storages) | export location
|
Storage Methods Not For VidiCore
For some scenarios storage methods need to be defined that are not intended to be used by VidiCore but for other modules of an Enterprise MAM Solution. For such methods set a storage method different to NONE
to prevent VidiCore from using it.
Such a storages method may be an additional storage method on top of the ones usable by VidiCore to model alternative access methods to the storage (which may be faster, more efficient, etc.).
There may also be storages which VidiCore cannot access at all (e.g. some video servers) and the non-VidiCore storage method is the only one available. In this case it has to be ensured that at all operations on this storage (e.g. housekeeping, copying files from and to the storage) are implemented by VidiFlow-based agents.
For deleting files it has to be ensured that not only the physical file but also the VidiCore FILE entity is properly deleted:
delete physical file with the storage’s / video server’s API;
unregister VidiCore FILE entity for that file;
if required, delete the VidiCore item.
The following method types are pre-defined an can be used in ConfigPortal:
Use-case | Storage Method Type in VidiCore | Allowed URI Schemes |
---|---|---|
Access to storage via VidiCore |
| all VidiCore-supported schemes |
Access to storage via UNC paths by Windows backend modules |
| UNC paths as |
Access to storage via mount points paths by Mac compontents |
|
|
Access to proxy storage by VidiStream |
| Any scheme accessible by VidiStream The storage method URI in VidiCore must not contain credentials. |
http-based up- and download by EditMate (e.g. media files, project files) or other products (but not VidiCore) |
|
|
Client-based access to a storage by EditMate: the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation |
|
|
Client-based access to a storage by Adobe Premiere: used for FCP XML export (e.g. by VidiEditor), the hires files are expected to be reachable via the configured URI from the Adobe Premiere workstation |
|
|
For referencing clips on Grass Valley Momentum |
|
|
Addressing a file via StorNext API, e.g. for moving a file from disk to tape. |
|
|