Project Nephos: Cloud based storage for a massive collection of TV recordings
Both CCExtractor and Red Hen (our sister organization you should check out) store massive archives of TV recordings. By massive we mean hundreds of terabytes. Until now these archives have been handled in-house, but we're approaching a point in which it's financially more sensible to use cloud storage.
During this summer we want to approach the migration to cloud storage, specifically using Google Drive (but your code should allow to extend to other services).
Some of the must-have features are easy, for example when a recording is complete (and exists as a local file) is needs to be moved to Cloud.
Other things will need more work. Specifically:
- Indexing. This is not a “index by date” or other trivial thing, we index by content.
- Sharing, which needs to be as flexible as possible. In general everything needs to be automatic. For example config such as “share all Spanish TV content with these American universities”, etc.
- Duplication, which means that content shared with us from another instance of Nephos can be copied to our own instance of Cloud storage.
- Pre and post processes, for example to convert the original format to smaller versions, or to extract subtitles.