Architecture of VidiCore - 29/01/2024
Introduction
In this blog post, we will explain what the VidiCore product is and how it fits into a modern IT design. The purpose is to give an introduction so that architects can get a first sense of how VidiCore can be deployed.
VidiCore. What is it?
What is VidiCore (VC)? VidiCore is a multi-service Java enterprise application, deployed using OCI on Docker or Kubernetes using Helm charts.
All interface to VidiCore is via the API. The VidiCore API is a RESTful web application, which supports several types of authentication mechanisms. The caller can use both XML and JSON as payloads, so it is easy to integrate. Typically, VidiCore instances are put behind a load balancer, which is also used for https termination.
The VidiCore application itself is modular, where the API can be on one node, and the business logic (or selected parts thereof) on another. VidiCore automatically finds the other nodes using JGroups.
All persistency – except for media – is in the SQL database. VidiCore uses the PostgreSQL database. For cloud deployments, we have very good results from using AWS RDS: both the native PostgreSQL service and the Aurora one. And speaking of Aurora, we also have deployments with the serverless edition, which scales really well. A typical production database size ranges from a few hundred GB up to a few TB. Remember that the VidiCore API allows you to really store a lot of information with each media clip.
To get good search performance, we use a search engine. We started out with Solr, but with AWS OpenSearch, we now have a service that is much more scalable, and easier to manage than the self-managed deployments. Internally between the components, a JMS message bus is used. VidiCore uses ActiveMQ, and for the cloud, some deployments have used Amazon MQ with good results.
If we combine the pieces above, we get something like the picture below. Remember that the number of VidiCore instances can be changed at any time, or which services a particular VidiCore instance should run.
We mentioned above that VidiCore is deployed both as OCI/Docker and K8s. On the cloud deployments, we have production systems both on AWS Elastic Container Service (ECS) and AWS Elastic Kubernetes Service (EKS).
Content is king and queen
VidiCore does not store the actual media for you, but it can manage storage. VidiCore supports dozens of storage protocols, including locally mounted SANs. For a cloud setup, S3 is by far the most popular choice, and we have spent a lot of time making a rich integration.
For example, VidiCore supports all storage tiers of AWS S3. This means that you can use the API to move content between different tiers, including Glacier, but are not forced to use the API. Life-cycle rules can also be used, and VidiCore will pick up the changes and make it visible to the VidiCore caller.
Worried that scanning a large S3 bucket will take a long time? No worries! VidiCore uses SQS notifications to get S3 updates so that storage rules, auto-imports, and notifications, immediately take effect. Likewise, VidiCore supports reading and writing S3 object tags, and make content available via CloudFront.
While S3 objects are immutable, VidiCore supports growing files on S3, using a chunk pattern. During the growth of the object, all API operations are available, including editing using VidiEditor (VE) or transferring the growing object further.
Making VidiCore exactly the way you want it and keeping it so.
Ok, we admit. Back in the early days VidiCore was not very easy to configure. You were at the mercy of the API. Fortunately, this has changed. ConfigPortal is a very powerful tool where you can configure storages, metadata schema, actually all important settings of VidiCore, in an intuitive user interface. ConfigPortal also supports versioning of configuration, so you can try out a configuration and safely revert if needed.
When it comes to user administration, users can be added manually, or an AD/LDAP integration can be set up. Or even better, use AuthService. AuthService is a service where several different authentication protocols are supported. AuthService is also easily configured with ConfigPortal.
For those who still want to use the API, we have developed an easy-to-use web application called Admin Tool. Admin Tool can be used with any VidiCore deployment and makes it easy to quickly do changes, check job and metadata status, and much more.
We also have some great news when it comes to ensuring the operation of VidiCore. VidiCore now supports the OpenTelemetry protocol (OTLP) for emitting vital information of the functioning of the different parts of VidiCore. This includes metrics such as running jobs, message queue length, and much more. It also supports tracing, which gives great insight into performance and where time is spent. Very useful if you are deploying VidiCore yourself.
Deployment models
It has always been important to us that VidiCore can be deployed the way you want it. Hence, we support a lot of deployment models. Below we list the five different categories of deployment.
OEM/package
Several partners are licensing VidiCore to build their own media solution. The artifact used there is the OCI image, and the partner is responsible for setting up the infrastructure and deploying VidiCore and all third-party components necessary. This model is primarily for VidiCore only.
On-premises
For customers who want to run everything in their own datacenter, VidiNet has all the tools. Terraform is used for infrastructure setup. For application deployment, we have a tool, PREPARED, which takes care of all dependencies and initial configuration. This model supports the largest datasets (tens of millions of assets, tens of thousands of users). But before you decide on the on-premises method because of your on-premises storage, check out VSA below.
SaaS
If you just want to try out VidiCore, or your dataset is medium-sized (let’s say up to a million of assets), we recommend one of our standard sizes of our Software-as-a-Service offering on http://vidinet.net . That way, you can have a running VidiCore instance in a matter of minutes, all while Vidispine personnel supervises system.
BYOC
Similar to the on-prem model, but on the cloud, is the Bring-your-own-cloud model. Here, VidiCore runs on your cloud account. This model scales to very large datasets, and by using scalable cloud services this scaling can be quite automatic. Also here, the PREPARED tool is used for deployment.
Managed BYOC
Interested in running on your own cloud, but you do not want to manage it? Vidispine is part of Arvato Systems. Arvato Systems (Asys) has a lot of expertise in operating services on customers’ clouds. Of course, you still have the flexibility to model the setup to your needs.
MediaPortal and VidiEditor
As mentioned above, the OEM model is the only model where our best-in-class tools MediaPortal (MP) and VidiEditor cannot be used. For all other models, MediaPortal is used to quickly manage and search your media repository, and VidiEditor is used to do browser-based editing and rendering. MediaPortal supports all of the richness of VidiCore’s data model and enhances it with integration to AI media services.
VidiEditor has a unique set of effects and transitions, and using the StreamingServer growing media can be edited in real-time. Both of these tools are installed by the PREPARED tool.
Getting the right format in time
While VidiCore can be used with many third-party transcoders, our own VidiCoder (VCR) transcoder is a very versatile converter which supports a lot of formats. It also supports transwrapping (changing the format but without recoding) and sequence rendering.
So VidiCoder can do the work, but we recommend to use the VidiNet Server Agent (VSA) as well. The VSA includes VidiCoder, but also comes remote deployment possibilities, support for more cloud storages, and also off-loads some work of VidiCore.
VSAs are deployed near your storage, so they can be deployed on-premises even if your VidiCore is running in the cloud, or on another cloud, or in another region. There are lots of possibilities. In addition to managing storage in a location-aware sense, it can also serve as an integration point for your other on-premise services.
VSA used to be a bit of a hassle to set up. This is no longer the case with ConfigPortal. With ConfigPortal, all VSAs can be individually configured, including their respective storage connections.- and VSA can be deployed on any container system: Docker/K8s/ECS/EKS.
Workflows. Connecting the dots.
While VidiCore has its own customizable job management, it does not come with a UI editor. If you are looking for a workflow engine that has an intuitive UI, and best of all: lots of scalable work agents specialized for media management, we recommend VidiFlow (VF). VidiFlow has built-in integration to MediaPortal and VidiEditor. VidiFlow is easily deployed in the cloud, for example by utilizing AWS EKS, RDS and Amazon MQ.
VidiNet Platform Services
In addition to VidiCore and VSA, the VidiNet Platform offers several media services as a service. These services are available to all deployments models, but work best when the media is stored on a cloud storage such as Amazon S3, Microsoft Azure Storage, or Google Cloud Storage. The media services include transcoding: both with VidiCoder and third-party transcoders such as Bitmovin Encoder, quality control by Baton, AI services by AWS and DeepVA, and VidiNet’s own optimized S3 object copy and hash compute services. By including these services, we might end up with an architecture that looks like the diagram below.
Summary
In this blog post, we have described some of the components of VidiNet, how they are deployed, and the third-party components used to build an enterprise-grade media asset management system. We have shown how the flexibility of VidiCore makes it possible to install VidiCore on-premises, in the cloud, or hybrid.
About the author
Isak Jonsson started his career with parallel computing and supercomputers. As a co-founder of Vidispine, he has played an important role of the implementation of an API first, cloud first, architecture.