Send VidiCore API service logs to an AWS S3 bucket [HT GEN]

Applies to

An environment where VidiCore API Service logs need to be stored at specified AWS S3 bucket location

Summary

This article describes how to set up VidiCore as-a-Service instance to log to an S3 bucket through the VidiNet Dashboard, including what permission a customer needs to set on their S3 bucket to allow the VidiCore instance to write its logs to the bucket. It also shows how to use LogStash to make use of the logs.

Prerequisites and system requirements needed for this article

An AWS account with the necessary control over the permissions needed for integration with VidiCore API / VidiNet.
A basic understanding of the AWS services such as S3 (the cloud storage) and IAM (the permissions service).
A verified installation of VidiCore API in VidiNet.

Applies to

An environment where VidiCore API Service logs need to be stored at specified AWS S3 bucket location

Summary

This article describes how to set up VidiCore as-a-Service instance to log to an S3 bucket through the VidiNet Dashboard, including what permission a customer needs to set on their S3 bucket to allow the VidiCore instance to write its logs to the bucket. It also shows how to use LogStash to make use of the logs.

Prerequisites and system requirements needed for this article:

An AWS account with the necessary control over the permissions needed for integration with VidiCore API / VidiNet.
A basic understanding of the AWS services such as S3 (the cloud storage) and IAM (the permissions service).
A verified installation of VidiCore API in VidiNet.

Setting up your AWS S3 bucket

To create an S3 bucket in AWS, direct your attention to the web page of your AWS account and the Service S3.

Edit the bucket policy on the created S3 storage. In the bucket policy editor, add the following policy for your bucket.

CODE

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "Statement3",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::823635665685:role/{vaas-name}-vidispine"
           },
           "Action": "s3:PutObject",
           "Resource": "arn:aws:s3:::{your-s3-bucket}/*"
       }
   ]
}

vaas-name is the first part of your VidiCore API endpoint:
Example API endpoint example.myvidispine.com would use vaas-name = example

For Partners

For Partners: that if you are a partner utilizing our staging environment a different ARN must be applied: "arn:aws:iam::763795643354:role/{vaas-name}-vidispine"

Configure Logging in VidiNet Dashboard

The log export is configured from the VidiNet Dashboard. Inspect your VidiCore service and go to the “MISC” tab. Enable log export by entering bucket name and a folder where you want the logs to be stored.

The logs will be stored in a folder hierarchy with the format yyyy/MM/dd/<logfiles>.

VidiCore as a Service will be recycled when applying or removing log export in VidiNet.

Logs will not be removed from S3 upon log destination setting removal in VidiNet.

This operations takes a couple of minutes, during that time your system will be in updating state. The Search index will be kept intact, and no further initialization is needed.

When you have created the log destination it will be visible in the panel, allowing you to edit it, or remove it at any time.

Note: Make sure you use different buckets and/or folders for your different Vidispine API instances to avoid mixing up logs in the log folders.

Logstash

Once this bucket is configured the logs will start coming into the bucket with the selected prefix. There will be a lot of small log files, which you can for example pick up and make use of with LogStash. The following config can be used to pick up all new files and save them to a local file: server.log, it will also delete processed files to keep your bucket clean. Note that you can use both temporary credentials with a session token or static credentials:

CODE

input {
  s3 {
    "access_key_id" => "AKIA..."
    "secret_access_key" => "keVM..."
    "bucket" => "my-s3-storage"
    "backup_to_dir" => "/tmp/processed"
    "region" => "eu-west-1"
    "prefix" => "vs-logs/"
    "interval" => 5
    "delete" => true
    "additional_settings" => {
      "follow_redirects" => false
    }
  }
}
 
filter {
  grok {
    match => { "message" => "\[%{GREEDYDATA:instance}\] %{IP:client} - - \[%{GREEDYDATA:timestamp}\] %{GREEDYDATA:text}" }
  }
  mutate {
    strip => "message"
  } 
}
 
output {
  stdout { }
  file {
    path => "server.log"
    codec => line { format => "%{message}"}
  }
}

Start LogStash and you can investigate your logs from there:

CODE

$ logstash -f config.json

Applies to

Summary

Applies to

Summary

Setting up your AWS S3 bucket

For Partners

Configure Logging in VidiNet Dashboard

Logstash

Related articles