The “How do I digitize/transfer/capture video tapes” quick info thread (reddit backup)

FYI this post is a backup of my reddit post. Now that reddit is pulling a tumblr I figured self hosting is in order.


Why a “How do I digitize/transfer/capture video tapes” thread?

I see this question asked on DataHoarder a couple times a week, mostly with the same good and bad answers. Hopefully this thread will serve the purpose of providing basic and in depth info and options for digitizing common video tapes. Will be asking mods to sticky this. I’ll be making updates if anyone has strong opinions or things to add.

Who are you?

I frequent this sub and spend a lot of time transferring tapes for various communities, here’s my setup: https://www.reddit.com/r/DataHoarder/comments/g2l5ow/video_archival_rack_build_one_year_update_more/

Basics

The way in which you digitize your tapes is going to depend on the tape format and the quality you wish to achieve. As with most things, you get what you pay for, and the higher the quality you desire the more you’re going to go down a rabbit hole of information.

This post will cover the common formats you’re likely to deal with: VHS, VHSC, HI8, Video8, Digital8, and miniDV.

Continue reading “The “How do I digitize/transfer/capture video tapes” quick info thread (reddit backup)”

ECS cloudwatch task error: “The specified log group does not exist”

This quick one is just for the googlers.

Saw an error bringing up tasks on a fresh ECS cluster and task run. The task definition included configuration to send logs to a cloudwatch log group.

"logConfiguration": {
    "logDriver": "awslogs",
    "options": {
        "awslogs-group": "web",
        "awslogs-region": "us-west-2",
        "awslogs-stream-prefix": "ecs"
    }
}

Unfortunately this is missing a not well documented flag:

"awslogs-create-group": "true",

Adding this argument, along with the correct permissions to create log groups, allows the task to create the group and send logs. The following permission is needed on the task execution role:

logs:CreateLogGroup

Full docs here:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

Boxee Box “Can’t connect to internet” fix, cloned Boxee services

The Boxee Box was a short lived but powerful set top box by D-link that was released 2010 and discontinued 2012.

All Boxee Boxes relied on an application server hosted by D-link at boxee.tv for periodic phone-home calls and service endpoints.

In June 2019 these application servers went down, resulting in all Boxee Boxes still in operation throwing “Can’t connect to internet” errors and all user profiles and apps going offline.

In August 2019 I released a small python Flask app, boxee-server-light, to replace the downed boxee.tv servers. This code was created by referencing an existing project by Jimmy Conner (cigamit, boxeed.in forums).

To use it, you’ll need to add DNS entries for all boxee application urls, pointing to the boxee-server-light application.

For example:

18.211.111.89  app.boxee.tv
18.211.111.89  api.boxee.tv
18.211.111.89  dir.boxee.tv
18.211.111.89  s3.boxee.tv
18.211.111.89  t.boxee.tv
18.211.111.89  res.boxee.tv
18.211.111.89  0.ping.boxee.tv
18.211.111.89  1.ping.boxee.tv
18.211.111.89  2.ping.boxee.tv
18.211.111.89  3.ping.boxee.tv
18.211.111.89  4.ping.boxee.tv
18.211.111.89  5.ping.boxee.tv
18.211.111.89  6.ping.boxee.tv
18.211.111.89  7.ping.boxee.tv
18.211.111.89  8.ping.boxee.tv
18.211.111.89  9.ping.boxee.tv
18.211.111.89  dl.boxee.tv

… where the IP is the address of the Flask application.

For those who are unable to run their own DNS or this application, I am hosting a public version of this code. You can add my public DNS server to your router config, or set it as custom DNS on your Boxee Box in network settings. You can also point directly to my public application server using your own DNS.

My public DNS server is by whitelist only, so please email me (I don’t check comments often) if you would like access.

Public DNS server address: 18.211.111.89
Public application server address: 18.213.38.199 18.211.111.89 (also)

For more up to date info and discussion, check out my Reddit post:
https://www.reddit.com/r/boxee/comments/ci4ugj/boxee_cloned_server_updates_working_server/

 

FAQ:

Do I need a static IP address from my ISP to use the public DNS?
Yes. I’ll need to whitelist your IP address. If you get a new one every day this won’t work.

I run my own local DNS. Do I need to be whitelisted to use your application server?
No. Map the boxee domains to my public app server as shown above. No whitelist required.

I’m logged out of my boxee box. How do I log back in while using this app?
Any username and password combo will work to log you back in.

I reset my boxee box. What firmware do I need to be using to use your public servers?
1.5.1 (latest) seems to work best. If you can’t find this firmware, email me.

Do apps work with this project?
I don’t have any apps connected yet. PRs welcome. I’m not 100% sure if app downloading will work without some additional code.



HAProxy dynamic backend selection with Lua script

HAProxy is a popular load balancer with extensive configuration options, including the ability to influence balancing and other options via Lua scripts.

In this post i’ll show how it’s possible to influence HAProxy backend selection via a Lua script. The use case for this situation was the necessity to choose a backend server based off of the responses from each possible backend.

First, Lua needs to be installed and HAProxy needs to be installed with Lua support. This involves building HAProxy with USE_LUA=1 environment var set during make.

Here’s a stripped down config example, with relevant lines commented. Not all required config attributes are included for brevity.

global
    # Load custom lua script. I usually put this alongside the haproxy.conf.
    lua-load /etc/haproxy/pick_backend.lua 

# Frontend config, rtmp traffic.
frontend frontendrtmp
    bind *:1935
    mode tcp

    # inspect-delay was required or else was seeing timeouts during lua script run
    tcp-request inspect-delay 1m

    # This line intercepts the incoming tcp request and pipes it through lua function, called "pick backend"
    tcp-request content lua.pick_backend

    # use_backend based off of the "streambackend" response variable we inject via lua script
    use_backend %[var(req.streambackend)]


# Example backends. One server per backend. The Lua script will iterate through all backends
# with "backendrtmp" prefix. 
# HAProxy use_server attribute does not yet support lua scripts, so backends necessary.
backend backendrtmp1
    mode tcp
    server rtmp 123.456.789.0:1935 check

Requests to the “frontendrtmp” frontend are routed through the Lua script, which checks each listed backend and chooses one based off its response.

Here’s the Lua script:

local function pick_backend(txn)
    winner_name = 'backendrtmp1' -- Needs to match available backend.
    winner_count = -1 ---initial count flag

    for backend_name ,v in pairs(core.backends) do
      if (backend_name ~= 'MASTER') then -- Filter out built in backend name
        -- iterate backend servers dict, assuming one server per backend.
        for server_name, server in pairs(v.servers) do
          -- Skip any server that is down.
          if server:get_stats()['status'] ~= 'DOWN' then
            address = string.match(server:get_addr(), '%d+.%d+.%d+.%d+')
            local tcp = core.tcp()
            tcp:settimeout(1)

            -- Connect to rtmp server to get stats counts.
            if tcp:connect(address, 80) then
              if tcp:send('GET /statistics\r\n') then
                local line, _ = tcp:receive('*a')

                -- Do whatever checks you want here with the response.
                -- In this case, i'll just check the number returned
                -- from the statistics endpoint.
                streamers = tonumber(string.match(line, '(%d+)'))

                -- Check and set winner.
                if (winner_count == -1) then
                  print('Set initial backend', backend_name)
                  winner_count = streamers
                  winner_backend = backend_name
                else
                  if (streamers < winner_count) then
                    print('New winner', backend_name)
                    winner_count = streamers
                    winner_backend = backend_name
                  end
                end
              end
              tcp:close()
            else
              print('Socket connection failed')
            end
          end
        end
      end
    end
    print('Winner is:', winner_backend)

    -- Set winner backend name to variable on the request.
    txn:set_var('req.streambackend', winner_backend)
end

core.register_action('pick_backend', {'tcp-req', 'http-req'}, pick_backend)

The Lua script:

  • Iterates over each backend with the required prefix
  • Hits an endpoint on the listed server if it's up
  • Checks count from endpoint
  • Compares with count of previous lowest count server
  • Sets a response variable with the name of the backend with the lowest count

This enables us to route traffic dynamically to the server with the lowest number of users.

Regression testing releases with Depicted (Dpxdt), Travis, & Saucelabs

Depicted is a release testing tool that compares before and after screenshots of your webpage, highlighting differences between the two.

Depicted supplements your release testing by allowing you to approve any visual changes a new release may cause.

I wrote a script during my time at Sprintly that would take a TravisCI build ID, pull related screenshots from our Saucelabs selenium tests, and upload them to a Depicted API server for comparison.

Before a new release would be deployed, we would manually run our Depicted release script and check and approve any changes.

This script was integrated as a Django management command for ease of use. Check out the full script below with comments.

Ubuntu server 12.0.4 SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Ran into some trouble with Ubuntu 12.0.4 and an “SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure” error while accessing the Zoho Docs API through Python. Apparently Zoho’s API endpoint enforces a certain SSL version. The full python error was:

SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

After ruling out python and openssl, the fix was to simply update to Ubuntu 12.0.4.1 through apt:

apt-get update
apt-get upgrade

And for the hell of it did a distribution upgrade:

apt-get dist-upgrade

python-cloudcracker: a class to interface with the CloudCracker API

So I released a quick python class to interface with the CloudCracker API. CloudCracker is a new online password cracking service. Upload a pcap file, pay via BitCoin or Stripe, and they will crack the password for you. Pretty cool. Check out CloudCracker at cloudcracker.com and check out python-cloudcracker on GitHub.

Django runserver and gevent; NotImplementedError: gevent is only usable from a single thread

Hey. So I ran into a problem when using Django runserver and Requests for Python. Specifically, when running runserver, I came across the error:

NotImplementedError: gevent is only usable from a single thread

To fix this, you need to run the gevent monkey patch in your projects manage.py.

Add the following lines in the __main__ of your manage.py

from gevent import monkey
monkey.patch_all()

Your main will look as follows:

if __name__ == "__main__":
    import gevent
    from gevent import monkey
    monkey.patch_all()
    execute_manager(settings)