Pain Points Require Improvements

However, there are still some pain points that exist in server-based VMS that need to be addressed.

1. Rebuilding a failed disk in disk array takes too much time. The risk of lost data increases with the size of the disk. No matter the VMS servers use an internal storage or external storage system, they use the RAID controller to protect data from loss.  If a disk drive fails, the disk array controller will take a very long time to rebuild the files to the spare drive. The rebuild time and the possibility of failure to rebuild drive are proportional to the size of the disk. Today, the largest hard drive is 16TB and sooner 20TB drive will be available. Using small size disks increases the total cost of storage servers.
2. The storage throughput suffers a big impact during the degrade rebuilding mode.
3. The traditional server has a large failure domain. If a server fails, all videos inside the server become inaccessible or even lost. More disks are installed in a server, more video files are lost at the same time when the server fails.
4. VMS lack for an elastic hardware architecture.

Use Software-Defined Storage and Arm MicroServer with Network Optics Nx Witness Video Management System

In this article, we will present how Ambedded's software-defined storage appliance and Arm-based microserver integrate Network Optics' VMS software to solve some of the issues that exist in the current solution.

Software-Defined Storage Appliance

Ambedded's software-defined storage Mars 400 Ceph Storage Appliance is a turn-key software-defined storage solution for enterprises. The appliance integrates ARM-Based microserver with storage software Ceph and web user interface. Mars 400 provides a unified storage cluster that supports the object, block, and file system. It is designed to provide excellent performance, reliability, and scalability with no single point of failure and minimum failure domain.

Nx Witness Video Management System & Nx Server

Nx Witness VMS is an open IP video management system(IPVMS) designed to find, view, record, and manage IP video cameras as well as integrate quickly with 3rd party systems and devices. The Nx Server application is a lightweight, powerful media server responsible for discovering, and managing Nx Witness system devices and data. Nx Server records video from IP cameras to inside or externally attached storage. It has server hive architecture ensuring no single point of failure.

Arm Microserver Architecture

The microserver platforms accommodate 8x Quad-Core ARM 64-bit servers modules in a one-rack-unit (1RU) chassis. Two in-chassis redundant switches provide 4x10 Gbps uplink for horizontal scale-out and client access. Every ARM microserver module is an independent Linux server that owns dedicated CPU, memory, storage, and network interface resources for application software.

The microserver architecture provides greater accessibility due to its lack of a single point of failure. Every microserver node can be configured as a storage node of a storage cluster or a VMS server node of the VMS cluster.

When the microserver is configured as a storage node, it joins the Ceph storage cluster to work with other storage nodes. If we configure the microserver as an Nx Server, it does his job as a VMS server and mounts a virtual disk from the Storage cluster for storing the recorded video files. This means that you can flexibly deploy the necessary number of Nx servers and Ceph storage servers all on Mars 400 microservers. Because of the advantages of microserver, you have the flexibility of scale the surveillance system with a small increment.

How Software-Defined Storage Protect and Reheal Data Against Hardware Failures

Ceph storage uses replication or erasure code algorithm to protect data from loss instead of the RAID controller. Erasure coding is analog to the RAID 5 or RAID 6 in the disk array but with much flexibility of allowing more disk fail currently. It can protect data not only from disk drive failures but also protect data against server failures, rack failure, and any scale of failure domain. Data or video files are first stripped to objects and every object is split to a number of data chunks and calculated redundant chunks. The storage server distributedly stores data and redundant chunks to the different storage servers in the storage cluster. If a storage server breaks, all data stored inside that server have to be rehealed. Data healing is done by all healthy servers and disks. And, only the lost chunks have to be re-healed instead of a rebuild all of the whole disks.

There are three major advantages of distributed software-defined storage compare to the RAID storage system.

1. Data can survive from one or more server node breaks.
2. Data reheal is much faster than RAID because all healthy disks in the cluster help data healing in parallel and only lost data need healing. If there are more disks in the cluster, healing speed is faster. An experiment of losing one 60% usage 10TB drive in a  36 drive cluster took 2.5 days for reheal. The reheal speed is 28.5MB/s for this experiment scale. (Clay Erasure code 4+2 profile)
3. During the reheal, healing does not interrupt the video recording
4. You can use large capacity disk drives such as 16TB HDD in the Ceph storage cluster to get the same capacity with fewer servers.

How Many Camera Recording can One Microserver Support

We set up a testing environment to stress the Nx Server on Ambedded Mars 400 microserver. The experiment firstly used a server to simulate IP cameras for streaming videos to an Arm microserver. The video stream format is H.264, 1080p high quality at 30 frames per second. The Arm VMS server mounts a virtual disk as its storage. The test result concludes that one microserver can record a maximum of 30 cameras without losing the video frame. The testing also includes the scenario while the virtual disk is full. This implies that a 1U Mars 400 that accommodates eight microservers can support 240 cameras high definition videos. This implies that a 1U Mars 400 that accommodates eight microservers can support up to 240 camera high definition video streams. This is a great achievement by using a 1U server and consumes only 100 Watts of power.

Test with Real IP Cameras

In another test, we use fifty IP cameras for making the test in a real environment.

Equipment list:

• IP Cameras: 25x Wired and 25x Wireless
• Mars 400Nx server: 8 nodes in 1U. 4 cores Arm A72, 4GB RAM & 2x 2.5Gbps LAN
• Nx Server: 3 Mars 400Nx Arm nodes,

♦ 3 nodes are active for recording and playback. There is a spare resource on each node for one node failover.

• Storage:

♦ Data protection: erasure code 4+2, allow up to 2 hard drive fails at the same time

♦ Ceph server: 21x microserver for Ceph OSD + 3x microserver for Ceph monitor

♦ Hard drives: 21x 16TB Seagate Exos Enterprise SATA HDD. 224TB is usable.

• Network:

♦ 4x 10Gb ports for 4x Mars 400Nx

♦ 1x Gb switch for cameras

♦ Wireless access point x 2

During recording and replay, only 150% of CPU resource is used. The maximum usable CPU resource is 400%. Please refer to figure 1.

Figure 1: Nx Server node CPU usage

There is a total of 4GB memory available in each node. 1GB memory is reserved for spare use. The memory amount used during the test is about 2.8 GB. The server node can stably support up to 30 cameras.

Figure 2: Nx Server node unused memory

The network traffic during the test is about 150Mbps download and 130Mbps for upload. The 2x 2.5Gb network bandwidth is much higher than the real traffic.

Figure 3: Nx Server nodes network traffic

VMS Server Node FailOver

Network Optics VMS server has the automatic server failover feature, which allows healthy VMS servers to connect cameras from a failed Nx VMS server. In our test, we set up three microservers as the Nx VMS server.  They can handle a total of 90 cameras. In the normal condition, 50 cameras under test are connected evenly to the three cameras. We simulate a server offline by turning it off. We prove 18 cameras connected to the failed server can automatically reconnect to the other two servers in about one minute. 

This implies that using the microserver solution for the Nx VMS server; you need only 1/8 of a 1U Mars 400Nx server spared for the failover. Compare to a big node server; microserver architecture can save one server cost for the failover setup.

Design example

Project requirements

1. Number of cameras: 200 cameras
2. Bit Rate per camera for 1080p, high quality, 30 FPS: 4.87Mbps
3. Recording hours per day: 16 hours
4. Days to record: 60 days

Calculation of server requirements

1. The total usable storage capacity required: 411 TB
2. The total raw capacity required if we use erasure code 4+2: 678 TB
3. Hard drive capacity: 16 TB
4. Number of hard disks required: 43 HDDs
5. Number of node required for storage (Ceph OSD): 43
6. Ceph Monitor node: 3
7. Nx server nodes for 200 cameras: 200/30 = 7 nodes.
8. Spare node for NxServer failover: 1 node
9. Total microserver nodes required: 43 + 3 + 7 + 1 = 54 nodes
10. Total Mars 400Nx required: 54/8 = 7x 1U Mars 40Nx
11. Spare nodes: (7 x 8) - 54 = 2 nodes
12. Total power consumption of server exclude HDD: 7 x 100 Watts = 700 Watts

Conclusion

Ambedded Mars 400Nx combines Nx VMS server, and software-defined storage has the following advantages:

1. Mars 400Nx's Software-defined storage can protect data against not only disk fails, but also the server fails.
2. When there are disk fails, Mars 400Nx re-healed data parallelly by all healthy drives. It is 4 to 5 times faster compared to the RAID controller.
3. Mars 400Nx can maintain its video recording and replay during data healing.
4. You can use large capacity disk drives such in the Mars 400Nx storage cluster to reduce the total cost of the storage system.
5. A 1U Mars 400Nx VMS server can support up to 240x 1080p cameras. You can design the VMS system with a flexible combination of storage and VMS servers.
6. Mars 400Nx Arm server saves 70 percent of power consumption compares to x86 servers.

About Ambedded

Ambedded is a software-defined storage company with expertise on Linux OS, kernel, software-defined storage, embedded system, and Arm server.

Ambedded's vision is to make the use of Ceph software-defined storage easy so enterprises can benefit from Ceph with a shorter learning curve.

Ambedded makes Ceph simple by offering easy to use user interface, ready to use storage appliance, stability & performance tuning, automate deployment, and Professional support.

With the purpose-built Ceph appliance, the Ambedded team owns wide experience to help customers to adopt our ceph solution into versatile industries, such as Telecom, medical, military, edge data center, and enterprises who require high availability storage.

Visit the Ambedded web site for details.

About Network Optics

Network Optix is a software development company focused on creating cutting edge video management solutions. Our core product - Nx VMS - is an open, lightweight, customizable video management platform capable of managing offline media, live video streams, and integrated 3rd party systems and devices. Nx VMS is a free live-streaming platform - cloud enabled, remotely accessible, scalable to any size, and licensed only when recording and search of IP Video cameras are required. Nx VMS is also open for developers, with comprehensive API and SDK documentation built-in and free to use, allowing organizations or individuals to create their own video-driven applications for any vertical market or project.

Acknowledgments

We would like to thank Seagate and Network Optix for providing the Exos X16 Enterprise Hard Drives and IP cameras used for this project.

Reference:

• Why RAID stop working in 2019, By Robin Harris for Storage Bits
  https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/

• Why the best RAID configuration is no RAID configuration by Guest Blogger → https://blog.shi.com/hardware/best-raid-configuration-no-raid-configuration/