Media Input/Output, Bundle and Composure

Media Input/Output, Composure, and Take Recorder Set Up

This section describes how to set up Media Profile and Media Bundle for use with ARwall ARFX Pro. This will add an input signal from the physical camera into Unreal Engine 5 to which live green screen compositing settings can be set and add an output signal of the final composited shot.

Important

Media Profile and Media Bundle options will vary depending on each project. This section will use Black Magic Media Source as an example, but make sure to match these options to the correct settings specified by input and output needs of your specific project.

Important

Support for Blackmagic Video is a feature created by Unreal Engine 5. Its official documentation can be found here: https://docs.unrealengine.com/5.1/en-US/blackmagic-video-io-quick-start-for-unreal-engine/. If using an AJA Video instead, the official documentation can be found here: https://docs.unrealengine.com/5.1/en-US/aja-video-io-quick-start-for-unreal-engine/

Setting Up Media Profile.                                              

This section describes how to set up the Media Profile to work with ARwall ARFX Pro in Unreal Engine 5.

1. Navigate to the Media Profile icon in Unreal Engine 5. Click on the arrow button to the right of the icon and select New Empty Media Profile.

2. A new window called Pick Media Profile Class should open. Select MediaProfile under All Classes and click Select button.

3. A new window called Save Asset As should open. Choose the folder location (can keep the same as the default location recommended by Unreal Engine 4) and name Media Profile with the desired name. Click the Save button.

4. A new window tab with the Media Profile name should open. Under Details tab,    

a. Under Inputs > Media Sources > MediaSource-01, click on None pull down menu and select correct media source. In example below, BlackMagic Media Source has been selected.

b. Under Inputs > Media Sources > MediaSource-01 > BlackMagic > Configuration, click the down arrow pull down menu and specify the following to match the hardware settings of the decklink and physical camera settings being used.

i. Device: Select the decklink input port that is connected to the physical camera’s SDI output via SDI cable.

ii. Resolution: Select the resolution that the physical camera is set to.

iii. Standard: Select whether the physical camera is set to Progressive or Interlaced setting.

iv. Frame Rate: Select the frame rate that the physical camera is set to.

v. Click Apply button when done.

BEST PRACTICE: Make sure to note the exact settings set under Inputs > Media Sources > MediaSource-01 > BlackMagic > Configuration as these same settings will need to be applied to other sections while setting up Media Inputs.

c. Under Inputs > Media Sources > MediaSource-01 > BlackMagic > Timecode Format, click None pull down menu and select the Timecode Format that matches the physical camera’s timecode format (LTC or VITC).

BEST PRACTICE: Make sure to note the exact settings set under Inputs > Media Sources > MediaSource-01 > BlackMagic > Timecode Format as these same settings will need to be applied to other sections while setting up Media Inputs.

d. Under Inputs > Media Sources > MediaSource-01 > Video > Color Format, click on the pull down menu and make sure to match the color format of the physical camera (8bit YUV or 10bit YUV).

e. Under Inputs > Media Sources > MediaSource-01 > Video > Is SRGBInput, recommended to select this box.

f. Under TimeCode Provider > Override Project Settings, select this box.

g. Under TimeCode Provider > Timecode Provider, click on None pull down menu and select correct media source. In example below, BlackMagic Media Source has been selected.

h. Under TimeCode Provider > Timecode Provider > Blackmagic > Media Configuration, click down arrow pull down menu and specify the following to match the hardware settings of the decklink and physical camera settings being used.

i. Device: Select the decklink input port that is connected to the physical camera’s SDI output via SDI cable.

ii. Resolution: Select the resolution that the physical camera is set to.

iii. Standard: Select whether the physical camera is set to Progressive or Interlaced setting.

iv. Frame Rate: Select the frame rate that the physical camera is set to.

v. Click Apply button when done.

BEST PRACTICE: Make sure these settings are the same as those set under Inputs > Media Sources > MediaSource-01 > BlackMagic > Configuration.

i. Under TimeCode Provider > Timecode Provider > Timecode Format, click None pull down menu and select the Timecode Format that matches the physical camera’s timecode format (LTC or VITC).

BEST PRACTICE: Make sure these settings are the same as those set under Inputs > Media Sources > MediaSource-01 > BlackMagic > Timecode Format.

j. Under Genlock > Override Project Settings, select this box.

k. Under Genlock > Custom Time Step, click on None pull down menu and select correct media source. In example below, BlackMagic Media Source has been selected.

l. Under Genlock > Custom Time Step > Genlock > Configuration, click down arrow pull down menu and specify the following to match the hardware settings of the decklink and physical camera settings being used:

i. Device: Select the decklink input port that is connected to the physical camera’s SDI output via SDI cable.

ii. Resolution: Select the resolution that the physical camera is set to.

iii. Standard: Select whether the physical camera is set to Progressive or Interlaced setting.

iv. Frame Rate: Select the frame rate that the physical camera is set to.

v. Click Apply button when done.

BEST PRACTICE: Make sure these settings are the same as those set under Inputs > Media Sources > MediaSource-01 > BlackMagic > Configuration and under TimeCode Provider > Timecode Provider > Blackmagic > Media Configuration.

m. Under Genlock > Custom Time Step > Genlock > Genlock options > Display Dropped Frames Warning, recommended to select this box if running a heavy scene in Unreal Engine 4.

5. Once all settings above have been set, click Apply icon and click Save icon at the top of the window. Close out Media Profile window.

6. Re-initialize timecode provider by clicking on the down arrow drop down menu and selecting Timecode Provider > Reinitialize.

a. If all settings are correct and physical camera SDI is plugged into the machine, the timecode in Unreal Engine 4 should match the timecode on the physical camera.

b. If Timecode Provider is not already showing in the Unreal Engine 4, select it under the main menu Window > Developer Tools > Timecode Provider to open up the Timecode Provider window.

BEST PRACTICE: Make sure that the above settings under Media Sources matches the physical hardware (decklink and physical camera) exactly. Otherwise, there will be issues receiving the input feed correctly in Unreal Engine 4. If settings are off, this can be confirmed under the Output Log in Unreal Engine 4 as error messages.

Section 8.2: Setting Up Media Bundle

This section describes how to set up the Media Bundle to work with ARwall ARFX Pro in Unreal Engine 4.

1. Navigate to Content Browser > Content folder, right click inside the folder window and select Media > Media Bundle.

2. A NewMediaBundle should show up in the Content folder. Double click on the NewMediaBundle and a NewMediaBundle window should open up.

3. Under Detail > Media Bundle > Media Source, click on None pull down menu and select correct media source. In example below, BlackMagic Media Source has been selected.

4. Under Details > Media Bundle > Media Source > Blackmagic > Configuration, click down arrow pull down menu and specify the following to match the hardware settings of the decklink and physical camera settings being used (these should match the same settings set under Blackmagic configurations for Media Profile):

a. Device: Select the decklink input port that is connected to the physical camera’s SDI output via SDI cable.

b. Resolution: Select the resolution that the physical camera is set to.

c. Standard: Select whether the physical camera is set to Progressive or Interlaced setting.

d. Frame Rate: Select the frame rate that the physical camera is set to.

e. Click Apply button when done.

BEST PRACTICE: Make sure these settings are the same as those set in section 8.1: Setting Up Media Profile under Inputs > Media Sources > MediaSource-01 > BlackMagic > Configuration, under TimeCode Provider > Timecode Provider > Blackmagic > Media Configuration, and under Genlock > Custom Time Step > Genlock > Configuration.

5. Under Details > Media Bundle > Media Source > Blackmagic > Timecode Format, click None pull down menu and select the Timecode Format that matches the physical camera’s timecode format (LTC or VITC). (These should match the same setting set under Timecode Formats for Media Profile.)

BEST PRACTICE: Make sure these settings are the same as those set in section 8.1: Setting Up Media Profile under Inputs > Media Sources > MediaSource-01 > BlackMagic > Timecode Format and under TimeCode Provider > Timecode Provider > Timecode Format.

6. Optional: Under Details > Media Bundle > Media Source > Synchronization > Synchronize with Engine’s Timecode can be clicked on. This setting may already be defaulted on and the engine is likely already running at the same frame rate.

7. Click Save icon and close out the NewMediaBundle window.

8. To test the NewMediaBundle that was just created, drag and drop the NewMediaBundle from Content Browser > Content > NewMediaBundle into the Unreal Engine 4 editor window.

a. Press Play on the Unreal Engine 4 scene.

b. Navigate to the Output Log window and look for any messages regarding LogBlackMagicMedia or whichever decklink card is being used.

c. Press Stop on the Unreal Engine 4 after checking the Output Log.

d. If NewMediaBundle has been set up correctly, a video signal should be seen in the NewMediaBundle viewframe in the editor. The NewMediaBundle should look like a frame of whatever is being captured by the physical camera in the Unreal Engine 4 editor window. If settings were not set correctly, the NewMediaBundle will look like this:

BEST PRACTICE: If the physical camera feed is not showing up, it is recommended to make sure that the physical camera is properly set up, refer to Section 5: Full Hardware Set Up. Also, review settings set above in this section and section 8.1: Setting up Media Profile and make sure all settings input into Unreal Engine 4 match the physical camera settings.

Section 8.3: Setting Up Media Output

This section describes how to set up the Media Output to work with ARwall ARFX Pro in Unreal Engine 4.

1. Navigate to Content Browser > Content folder, right click in the folder and select Media > Blackmagic Media Output.

2. A NewBlackMagicMediaOutput should show up in the Content folder. Double click on the NewBlackMagicMediaOutput and a NewBlackMagicMediaOutput window should open up.

3. Under Details > Blackmagic > Configuration, click down arrow pull down menu and specify the following to set the preferred output (these do not need to match the input settings that were set up for Media Profile or Media Bundle, and can be any format that is the desired output):

a. Output Type: Fill

b. Device: Select the Decklink SDI output port that is connected to a preferred external capture or broadcast device.

c. Destination: Single Link 0

d. Resolution: Select preferred output resolution.

e. Standard: Select preferred Progressive or Interlaced setting.

f. Frame Rate: Select preferred output frame rate.

g. References: Free Run or External.

h. Click Apply button when done.

4. Under Details > Blackmagic > Synchronization > Wait for Sync Event, click on this box to make sure that everything being outputted is synced with the timecode of the camera.

5. Click Save icon and close out the NewBlackMagicMediaOutput window.

Section 8.4: Adding Composure

This section describes how to add composure (Unreal Engine 4’s green screen compositing tool) to work with ARwall ARFX Pro in Unreal Engine 4.

NOTE: Composure is a feature created by Unreal Engine 4. Its official documentation can be found here: https://docs.unrealengine.com/en-US/WorkingWithMedia/Composure/QuickStart/index.html

1. Under Window Menu, select Composure Compositing. This should open up a Composure Compositing window.

2. In the Composure Compositing window, right-click anywhere and select Comp Shots > Create New Comp.

a. A new window called Pick an Comp Class should open up. Click on Empty New Comp button under Common Classes.

b. A new comp should show up under Composure Compositing window. Keep name as is (0010_comp) or rename Comp as desired. This is an empty new comp.

3. Add CG Element and reference ARwall Canon Camera Actor.

a. Right-click on the new empty comp under Composure Compositing window, and select Layer Elements > Add Layer Element.

b. A new window called Pick an Element Type should open up. Click on CG Layer button under Common Classes.

c. New CG Layer (cg_element) should now be a child under the new comp (0010_comp).

d. With CG Layer selected under Composure Compositing window, navigate to Details > cg_element (self) > Composure > Input > Camera Source. Select Override.

e. Navigate to Details > cg_element (self) > Composure > Input > Target Camera Actor, click on the None pull down menu and select ARwallCanonCameraActor.

f. The comp now is referencing the desired Camera Actor as its CG element/plate.

4. Add Media Plate and connect to New Media Bundle.

a. Right-click on the new comp (0010_comp) under Composure Compositing window and select Layer Elements > Add Layer Element.

b. A new window called Pick an Element Type should open up. Click on Media Plate button under Common Classes.

c. New Media Plate (media_plate) should now be a child under the new comp (0010_comp).

d. With Media Plate selected under Composure Compositing window, navigate to Details > media_plate (self) > Composure > Inputs > MediaSource > Media Source. Under None pull down menu, select T_NewMediaBundle_BC media texture.

e. The comp is now referencing the New Media Bundle as its physical camera green screen element/plate.

5. Key the green screen plate using Composure.

a. Navigate to Details > media_plate (self) > Transform/Compositing Passes > Transform Passes.

b. Use preferred options under Chroma Keying (Multi Pass Chroma Keyer recommended), Despill (Multi Pass Despill recommended), and Erode to key out the green from the physical green screen plate. Please refer to Unreal Engine 4’s official documentation on further instructions on how to use these options.

6. Set Compositing Material.

a. Navigate to Composure Compositing and click on 0010_comp.

b. Navigate to Details > 0010_comp (Instance) > Composure > Transform/Compositing Passes > Transform Passes and click on + to add an array element.

c. Navigate to Details > 0010_comp (Instance) > Composure > Transform/Compositing Passes > Transform Pass_0 > Material and click on None pull down menu and select M_VPComp.

d. Navigate to Details > 0010_Comp (Instance) > Composure > Transform Passes > TransformPass_0 > Input Elements and select media_plate1 for Media Plate and cg_element1 for BG.

NOTE: For the example given here, the physical green screen plate (media_plate1) is layered in front of the virtual background (cg_element1), but any order of layers can be applied to Input Elements > MediaPlate and Input Elements > BG depending on the desired final output.

BEST PRACTICE: When playing a scene, be sure to select 0010_comp in the World Outliner to view the physical green screen plate (media_plate1) and virtual background (cg_element1) layered together in its final output.

7. Set Output for Comp.

a. Navigate to Composure Compositing and click on 0010_comp.

b. Navigate to Details > 0010_comp (Instance) > Composure > Output > Outputs and click on + to add an array element.

c. Under Details > Composure > Output > Outputs > OutputPass_0 > Capture Output, click on None pull down menu and select NewBlackMagicMediaOutput.

Section 8.5: Setting up and Using Take Recorder

This section describes how to set up and use Take Recorder (Unreal Engine 4’s tool that can record the position and rotation of anything that is in a scene) to work with ARwall ARFX Pro in Unreal Engine 4. Recording a take using Take Recorder allows the tracking information from a live shoot to be recorded, used later to re-render a different or altered CG background/scene, and the re-rendered background plate can then be re-synced with the original physical green screen plate footage. Using Take Recorder removes the need to use Unreal Engine 4’s Composure for green screen keying and allows users to do keying in post on their preferred platform.

NOTE: Take Recorder is a feature created by Unreal Engine 4. Its official documentation can be found here: https://docs.unrealengine.com/en-US/AnimatingObjects/Sequencer/Workflow/TakeRecorder/UsingTR/index.html

1. Set up Take Recorder.

a. Under Window Menu, select Cinematics > Take Recorder. This should open up a Take Recorder window.

b. Add the sources to record.

i. In the Take Recorder window, click on + Source pull down menu, and select From Actor > ARwallCameraRig to add the Camera Rig actor. This will record the physical camera tracking position and rotations.

ii. Click on + Source pull down menu, and select From Actor > ARwallCameraRig > ARwallCanonCameraActor to add the Canon Camera. This will record physical camera zooms and focus changes.

iii. Click on + Source pull down menu, and select From Actor > BP_SceneCenter to add the Scene Center.

iv. Clicking on ARwallCameraRig, BP_SceneCenter, or ARwallCanonCameraActor, will reveal Recorded Properties at the bottom of the Take Recorder window, where properties can be selected or removed from recording. It is recommend to keep the default selected properties as they should cover everything needed with using ARFX Pro.

2. Using Take Recorder.

a. While an Unreal Engine 4 scene is playing, navigate to the Take Recorder window and click the Red Record icon button in the upper right-hand corner of the Take Recorder window.

i. This will initiate a 3 second countdown overlay on the editor screen before Take Recorder begins to record.

ii. A recording will begin under Sequencer, actively tracking the selected sources from #1b.

b. To stop recording, navigate to the upper right-hand corner of Take Recorder window and click on the Stop icon button, or navigate to the bottom right hand corner of Unreal Engine 4 and click on the Stop button in the Take Recorder notification box.

BEST PRACTICE: Before accessing a recording, be sure to stop any scene that is playing in Unreal Engine 4, otherwise Unreal Engine will likely crash when opening up the recording.

c. To access the recording, navigate to Content Browser > Content > Cinematics > Takes > [today’s date] folder, and all recorded takes will be located here.

i. Double clicking the Level Sequence (Slate name and take number, ie. Scene_1_01) will open the take in the Sequencer window.

ii. Press Play button on the Sequencer to see the recorded sources in action.

iii. When Sequencer is open with a pre-recorded take, a copy of ARwall Camera Rig, ARwall Canon Camera Actor, and Scene Center (and any other sources that were added to Take Recorder to record) will appear in the World Outliner. These copies will have an orange lightning bolt on their icons.

NOTE: If Timecode is running correctly in Unreal Engine 4, Take Recorder will capture the timecode in its recording, making final compositing easy to sync up with the physical green screen plate footage.

d. After making changes to a scene in Unreal Engine 4, to re-record the background using the recording from Take Recorder, do the following:

i. Navigate to Composure Compositing window and click on the cg_element.

ii. Navigate to Details > cg_element (self) > Composure > Input > Target Camera Actor, click on pull down menu and select the new copy of ARwallCanonCameraActor (with the orange lightning bolt in its icon).

iii. Navigate to Sequencer and press Play button inside the Sequencer (make sure not to press the Play button for Unreal Engine 4 editor as this will cause a crash). This will play back the recording for the updated CG plate/background and push this to Unreal Engine 4’s output settings.

iv. A playback view finder called cg_element1 will show up in the Unreal Engine 4 editor window.

NOTE: As long as all assets/actors in an Unreal Engine 4 scene work in real-time, pressing play in the Sequencer should capture the newly updated CG plate/background.

BEST PRACTICE: Before creating a new take using Take Recorder, make sure to close out the Sequencer window. Closing the Sequencer window also removes the copy of the sources with the orange lightning bolts in their icons. Otherwise, Unreal Engine 4 will likely crash when playing a scene in the editor as it will be confused which Camera Rig, Canon Camera Actor, and Scene Centers should be used for playback.

v. After closing Sequencer, select cg_element under World Outliner, then navigate to Details > cg_element (self) > Composure > Input > Target Camera Actor and notice that this input is now set to None. Make sure to change it back to the original ARwallCanonCameraActor (without the orange lightning bolt in its icon) before playing a scene in the Unreal Engine 4 editor window. Otherwise, there will be no CG plate referenced for the Composure output.