MiniDV to Backup

Goals

  • Import all tapes onto external hard drive
  • Convert files to smaller format with very limited loss of quality
  • Backup to SpiderOak

Setup

Materials
Samsung Mini-DV camcorder model SCD80
IEEE.1394 Firewire cable
Linux Mint

How to import Mini-DV over IEEE-1394 Firewire on Linux Mint

My firewire card/port were already recognized by Mint. No special setup. Just had to connect camcorder to IEEE.1394 port via firewire.
Load tape, power on camcorder.

Used Synaptic Package Manager to install Kino
sudo kino (kino must run as root to see firewire devices since "someone at Mint/Ubuntu thinks firewire devices are more dangerous than USB devices." :-)
Edit > Prefs > 1394 > see the camera? Good.
~> Capture > Raw DV, Auto Split Files, write every 1 frame
OK

Capture Tab on right
rewind to Start of Movie
Capture button
verify files are being created

Testing

Convert dv file to various formats and compare an extracted screenshot

Extracted frames for comparison using
ffmpeg -sameq -i ../kino-capture-01.007.avi -f image2 %04d.avi.png
Then zoomed in and cropped lower left corner using Gimp.

original dv file imported by Kino
168M Jun 16 19:50 kino-capture-01.avi007.dv
0875.excerpt.dv.png

ffmpeg -sameq -i kino-capture-01.avi007.dv kino-capture-01.007.mpg
83M Jun 16 19:55 kino-capture-01.007.mpg
49% of original file size
0875.excerpt.mpg-sameq.png

ffmpeg -sameq -i kino-capture-01.avi007.dv kino-capture-01.007.avi
68M Jun 16 19:59 kino-capture-01.007.avi
40% of original file size
0875.excerpt.avi.png

ffmpeg -i kino-capture-01.avi007.dv kino-capture-01.007.mpg
3.8M Jun 16 19:57 kino-capture-01.007-default.mpg
2% of original file size
0875.excerpt.mpg-default.png

Test Results

Loaded all excerpts into Gimp as layers, zoomed in, and toggle compared to sort. Order starting with best…

  • avi format looked the best to me. It was slightly smoother than the mpg and dv. In a good way, not a dulled way.
  • mpg (sameq) format was pretty close, but considering the resulting file is bigger, why would I use it over avi?
  • dv format was third best in my opinion. Strange, since this was the source file. Perhaps the odd result is related to the decoding libraries being used while extracting the png frame.
  • mpg (default) was a distant fourth. Not feasible for my purposes, but good to keep in mind when I can take a quality hit for such a huge compression ratio.

Expected workflow
Import tapes witih Kino
Prefs > Capture > File: /root/vidcap/200711215-20071209-
Capturing on local drive first. Will move to external USB drive later. Doing this to reduce chances of dropped frames. Might not be necessary, but doing it anyways.
Kino will append 00n.dv resulting in files like
200711215-20071209-001.dv
200711215-20071209-002.dv
200711215-20071209-003.dv

Convert dv files to avi (ffmpeg -sameq)
Move converted files to folder protected by SpiderOak

File Name Clean-up, Conversion Testing

Had some old files originally imported with Vegas Movie Studio on Windows. Despite the avi extension, these were large files that would need to be converted/compressed. The file name convention I used at the time was

20020914-20021215 - Clip 001.avi

Before converting/compressing these old files, I wanted to clean-up the format to match what I've been using more recently. This would entail removing the spaces and the word "Clip." There has to be a more elegant way to do this, but the following worked.

Double check my use of awk:

ls | awk '{print($1"-"$4)}'
ls | awk '{print("mv \""$0"\" "$1"-"$4)}'

Then clean up the file names:

ls | awk '{print("mv \""$0"\" "$1"-"$4)}' | /bin/sh

Test convert one of the old avi files imported by Sony Vegas Movie Studio on Windows

for f in 20020914-20021215-057.avi; do ffmpeg -sameq -i "$f" "$f-compressed.avi"; done

while in the folder ~/HomeVideoOriginals/conversion-practice/ used the command:

for f in *.avi; do ffmpeg -sameq -i "$f" "../../HomeVideoConverted/conversion-practice/$f"; done

To convert file names with newer format of 20040206-20040208-037.dv, use the command:

single:
ffmpeg -sameq -i ./20040206-20040208-001.dv $HOME/HomeVideoConverted/20040206-20040208/20040206-20040208-001.avi

batch practice:
for f in 20040206-20040208-001.dv; do ffmpeg -sameq -i "$f" $HOME/HomeVideoConverted/20040206-20040208/$f.avi; done

batch do:
for f in *.dv; do ffmpeg -sameq -i "$f" $HOME/HomeVideoConverted/20040206-20040208/$f.avi; done

This results in a junky file name such as: 20040206-20040208-001.dv.avi which can be cleaned up with:
Test: ls | awk 'BEGIN { FS = "." }{ print($1"."$3) }'
Test: ls | awk 'BEGIN { FS = "." }{ print("mv \""$0"\" "$1"."$3) }'
Do it: ls | awk 'BEGIN { FS = "." }{ print("mv \""$0"\" "$1"."$3) }' | /bin/sh

Command Summary

For directories containing files with format "20041209-20041224 - Clip 001.avi"
ls | awk '{print("mv \""$0"\" "$1"-"$4)}' | /bin/sh
for f in *.avi; do ffmpeg -sameq -i "$f" "$HOME/HomeVideoConverted/20000000-20000000/$f"; done

For directories containing files with format "20040206-20040208-037.dv"
for f in *.dv; do ffmpeg -sameq -i "$f" $HOME/HomeVideoConverted/20000000-20000000/$f.avi; done
cd $HOME/20000000-20000000
ls | awk 'BEGIN { FS = "." }{ print("mv \""$0"\" "$1"."$3) }' | /bin/sh

Audio Gradually Out of Sync?

Lesson (re)learned: Evaluate and test output files very closely before converting all of your source media files.

Watching the converted video through the end revealed the audio track was increasingly out of sync with the video track as the clip progressed. Same behavior across different players.

From superuser.com: Almost always, when audio and video are out of sync and difference between the two get progressively worse over the course of the movie, especially after a conversion, the issue is one where the audio and video components were of a different length. As was observed in this instance, this happened because video frames were getting dropped, but the audio was relatively untouched during the conversion.

At times like this, it is best to first check the source movie to ensure that the audio was indeed in sync. If it is, then demux the movie, and deal with the video and audio components separately.

From simonbrenner.org:

Get information about a media file:

ffmpeg -i video.avi

Extract (demux) the audio track from a video file:

ffmpeg -i video.mts -acodec copy -vn audio.ac3

Extract (demux) the video track from a video file:

ffmpeg -i video.mts -vcodec copy -an -f h264 ffNT.h264

Merge an audio and a video track (mux):

ffmpeg -i video.h264 -i audio.ac3 -vcodec copy -acodec copy videoaudio.ts

Demux Audio And Video Then Merge

Well, this is what I need to test next.

ffmpeg -i 20020914-20021215-006.avi -acodec copy -vcodec copy 20020914-20021215-006.mkv
keeps audio in sync, same file size

ffmpeg -i 20020914-20021215-006.avi -acodec copy -vcodec libx264 20020914-20021215-006-vcodec-libx264.mkv
keeps audio in sync, slightly smaller file size

ffmpeg -i 20020914-20021215-006.avi -acodec copy -vcodec libx264 20020914-20021215-006-vcodec-libx264.avi
keeps audio in sync, much smaller file size, need to extract/zoom some frames to compare quality
Video playback in Totem was mottled grey, VLC worked fine

ffmpeg -sameq -i 20020914-20021215-006.mkv 20020914-20021215-006-mkv.avi
received many errors similar to: "[dvvideo @ 0x22f82c0] AC EOB marker is absent pos=65"
File was smaller, but video falling behind audio sync

Will continue testing…

Check into -asyncts and vsync

Using following original test file
476M Aug 9 2009 origtestfile.avi

john@asusv3 ~/Videos $ ffmpeg -i origtestfile.avi
ffmpeg version 0.10 Copyright (c) 2000-2012 the FFmpeg developers
built on Feb 14 2012 14:57:55 with gcc 4.6.2
configuration: —prefix=/usr —extra-cflags='-Wall -g ' —cc='ccache cc' —enable-shared —enable-libmp3lame —enable-gpl —enable-nonfree —enable-libdirac —disable-decoder=libdirac —enable-libvorbis —enable-pthreads —enable-libfaac —enable-libxvid —enable-postproc —enable-x11grab —enable-libgsm —enable-libtheora —enable-libopencore-amrnb —enable-libopencore-amrwb —enable-libx264 —enable-libspeex —enable-nonfree —disable-stripping —enable-libschroedinger —disable-encoder=libschroedinger —enable-version3 —enable-libopenjpeg —enable-libvpx —enable-librtmp —enable-avfilter —enable-libfreetype —enable-libvo-aacenc —disable-decoder=amrnb —enable-libvo-amrwbenc —enable-libaacplus —libdir=/usr/lib/x86_64-linux-gnu —disable-vda —enable-libdc1394 —disable-altivec —disable-armv5te —disable-armv6 —disable-vis —shlibdir=/usr/lib/x86_64-linux-gnu
libavutil 51. 34.101 / 51. 34.101
libavcodec 53. 60.100 / 53. 60.100
libavformat 53. 31.100 / 53. 31.100
libavdevice 53. 4.100 / 53. 4.100
libavfilter 2. 60.100 / 2. 60.100
libswscale 2. 1.100 / 2. 1.100
libswresample 0. 6.100 / 0. 6.100
libpostproc 52. 0.100 / 52. 0.100
Input #0, avi, from 'origtestfile.avi':
Metadata:
TAPE : 20090801-20090808
TCOD : 24683325333
TCDO : 25994635333
VMAJ : 4
VMIN : 0
CMNT :
RATE :
STAT : 3930 0 3.361859 1
DTIM : 0 0
Duration: 00:02:11.13, start: 0.000000, bitrate: 30410 kb/s
Stream #0:0: Video: dvvideo (dvsd / 0x64737664), yuv411p, 720x480 [SAR 8:9 DAR 4:3], 29.97 tbr, 29.97 tbn, 29.97 tbc
Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, 2 channels, s16, 1536 kb/s

Test 1:
ffmpeg -sameq -vsync -1 -i origtestfile.avi testfile-01.avi
Different behavior noticed was a flurry of "AC EOB marker is absent" error messages just before the command finished, instead of sprinkled throughout the command run time.
Resulted in much smaller file
173M Aug 12 19:40 testfile-01.avi

Audio seems to be in sync
Interlacing artifacts seem a little more noticeable in the new version

Test 2:
Similar to test 1, but try out the yadif (yet another de interlacing filter) to see if it reduces the interlace effect
ffmpeg -sameq -vsync -1 -filter yadif -i origtestfile.avi testfile-02.avi
Again, flurry of "AC EOB marker is absent" error messages at end of run.
Again, smaller file size
173M Aug 12 19:52 testfile-02.avi
Interlace effect seems the same as test 1

Test 3:

ffmpeg -sameq -vsync -1 -filter yadif=0:-1:0 -i origtestfile.avi testfile-03.avi
Same flurry, same size
173M Aug 12 19:59 testfile-03.avi

Suspending additional testing at this time due to time constraints. Primary goal of this effort is to get an off site backup copy of sentimental home movies. Since will be keeping original tapes and files, less than perfect quality is acceptable as long as the audio and video are in sync which appears to have been solved with the vsync option.

ffmpeg -sameq -vsync -1 -i origtestfile.avi testfile-01.avi

Command Summary (Keep Audio/Video in sync)

For directories containing files with format "20041209-20041224 - Clip 001.avi"
### Don't need to do this again ### ls | awk '{print("mv \""$0"\" "$1"-"$4)}' | /bin/sh
for f in *.avi; do ffmpeg -sameq -vsync -1 -i "$f" "$HOME/HomeVideoConverted/20000000-20000000/$f"; done

For directories containing files with format "20040206-20040208-037.dv"
for f in *.dv; do ffmpeg -sameq -vsync -1 -i "$f" $HOME/HomeVideoConverted/20000000-20000000/$f.avi; done
cd $HOME/20000000-20000000
ls | awk 'BEGIN { FS = "." }{ print("mv \""$0"\" "$1"."$3) }' | /bin/sh

What Worked Best So Far?

for f in *.avi; do ffmpeg -sameq -i "$f" "$HOME/HomeVideoConverted/20000000-20000000/$f.mpg"; done
ls | awk 'BEGIN { FS = "." }{ print("mv \""$0"\" "$1"."$3) }' | /bin/sh

Smaller file, audio in sync, reasonably good quality.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License