Serverless Smart Media Streaming Engine with AI/ML Backend

by React

Serverless Smart Media Streaming Engine with AI/ML Backend

Overview

This demo shows how you can use CDK and amplify to build a streaming service with a smart backend that analyze the streaming media using a AI/ML service. In this particular demo, it sends a soccer game that Tottenham plays, generate clips, and services only clips that Son Heung-min appears.

  • Elemental Media Services for streaming and vod contents
  • Serverless Event-Driven Architecture
  • All infra-structure resources as a code using CDK
  • Simple React frontend using Amplify
  • All code written in typescript

Architecture

Live Streaming

The streaming feature leverages Amazon Elemental Services. Amazon Elemental MediaLive provides live video processing service and Amazon Elemental MediaPackage provides packaging and distributions. And MediaPackage is integrated with the Amazon CloudFront to deliver live stream to viewers with low latency and high transfer speeds.

VoD Generation Backend with Rekognition

This demo shows a good demonstration on how you can use various types of events when developing serverless architecture. Amazon Elemental MediaLive has two output groups, one for MediaPackage for live stream and another for S3 Archive. The S3 Archive output group archives live videos into Amazon S3 bucket which becomes the first event of Event-Driven Serverless backend. When a media file is uploaded to S3 Archive Bucket, a Lambda is invoked to put a message including the object information into SQS queue. Then another lambda is called via SQS target, to call Amazon Elemental MediaConvert to convert the file. MediaConvert also has two output groups. One output group converts the file into HLS so it can be served as VoD Content. The other output group converts the file into MP4, because Amazon Rekognition can only analyze MP4 files. Amazon EventBridge watches the jobs of MediaConvert, and it triggers an event when it completes and calls the third lambda. The third lambda calles Amazon Rekognition to analyze who appears in the video clip. And if a certain player is found (“Son Heung-min” in this demo), it puts the metadata in Amazon DynamoDB.

Frontend Integration

For ease of configuration and deployment, this demo uses AWS Amplify for the frontend and AWS CDK for backend. To keep the infrastructure management simple and consistent, all the resources are defined and deployed using AWS CDK. And the resource information is exported into a file in JSON format, and the frontend imports that file to configure AWS Amplify resources. Detailed instruction will be provided in the later chapter.

Prerequisites

  • aws account
  • aws-cli
  • typescript >= 4.8
  • aws-cdk >= 2.54
  • node >= 16.17.0
  • npm >= 8.15.0
  • docker
  • git

Installation

  1. Download the code
git clone https://github.com/atheanchu/MediaReplayEngine.git
  1. Install Packages for CDK backend
npm install
  1. Export AWS CLI
export AWS_PROFILE=<your aws account profile>
  1. Bootstrap and synthesize CDK

cdk bootstrap
cdk synth
  1. Deploy CDK – export the output to the output file in frontend directory.
cdk deploy -O ./frontend/src/cdk-exports.json
  1. Install packages for frontend React application

cd ./frontend
npm install

Testing the demo

For detailed instructions on how to use OBS to stream your local video to MediaLive, please check this link.

  1. Install Open Broadcaster Software(OBS)

  2. Start MediaLive Channel

  3. Run OBS

  4. Click Settings and Streams. Choose Custom for Service.

  5. Copy MyMediaLiveChannelDestPri from ./frontend/cdk-exports.json. The value must be in the format of rtmp://<ip address>:<port>/<stream key>. For Server, input rtmp://<ip address>:<port>/ and for Stream Key, put <stream key>. For instance, if MyMediaLiveChannelDestPri is rtmp://555.555.555.555:9999/streamkey,

    - Server: rtmp://555.555.555.555:9999/
- Stream Key: streamkey

  6. Play a video that Son Heung-min is in. It could be a local file or youtube video.

  7. From Source, click + button and choose Window Capture. Select the window you want to start streaming to MediaLive input and click the button Start Streaming

  8. Run the frontend application

cd ./frontend
npm start
  1. It will show the streaming video at the top of the page. It takes a few minutes for the backend to generate clips and analyze each clips which is 60 seconds long. Click the refresh icon at the bottom right corner, and all the clips that a certain celebrity is in will show up below.

Clean Up

  1. Stop Streaming from OBS by clicking button Stop Streaming.
  2. Stop MediaLive Channel by clicking Stop Channel button.
  3. Since all the resources are deployed using AWS CDK, you can clean up simply by running cdk destroy. Although this demo uses AWS Amplify, we don’t have to run amplify delete because no resources are created using through Amplify.
cdk destroy

Resources

License

This sample code is made available under MIT-0 license. See the LICENSE file.

GitHub

View Github