This is article gathers notes taken from MapR's ADM 201 class which is mainly about:
- Testing & verifying Hardware before installing MapR Hadoop
- Installing MapR Hadoop
- Benchmarking MapR Hadoop Cluster configure new cluster for production
- Monitoring Cluster for failures & performance
Test disk performance
The disk-test.sh script checks the disk health and performance (i.e. throughput for sequential and random I/O read/write), it destroys any data available on it.
$ clush -ab /root/pre-install/disk-test.sh
For each scanned disk there will be a result file of the form disk_name-iozone.log.
First, download installer script
$ wget http://package.mapr.com/releases/v4.1.0/ubuntu/mapr-setup
$ chmod 755 mapr-setup
$ ./mapr-setup
Second, configure the installation process (e.g. define data and control nodes). A sample configuration can be found in /opt/mapr-installer/bin/config.example
$ cd /opt/mapr-installer/bin
$ cp config.example config.example.original
Use following commands to find information on nodes to declare in the configuration
$ clush -a lsblk # list drivers name
$ clush -a mount # list ip addresses and mounted drivers
Edit config.example file
$ ./install --cfg config.example --private-key /root/.ssh/id_rsa -u root -s -U root --debug new
In case the installation succeeded, you can login to https://master-node:8443/ with mapr:mapr to access MapR Control System (MCS) then get a new license.
Otherwise, if the installation fails, then remove install folder then check installation logs that can be found at /opt/mapr-installer/var/mapr-installer.log. Example of failures may be caused by:
$ rm -r -f /opt/mapr/ # remove installation folder
$ dpkg --get-selections | grep -v deinstall | grep mapr
mapr-cldb install
mapr-core install
mapr-core-internal install
mapr-fileserver install
mapr-hadoop-core install
mapr-hbase install
mapr-historyserver install
mapr-mapreduce1 install
mapr-mapreduce2 install
mapr-nfs install
mapr-nodemanager install
mapr-resourcemanager install
mapr-webserver install
mapr-zk-internal install
mapr-zookeeper install
$ dpkg -r --force-depends # remove all listed packages
To check if the cluster is running properly, we can run the following quick test job.
Note: check that the names of cluster nodes are resolvable through DNS, otherwise declare them in the /etc/hosts of each node.
$ su - mapr
$ cd /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
$ yarn jar hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi 8 800
First, copy the post-install folder to all nodes
$ clush -a --copy /root/post-install
$ clush -a ls /root/post-install
Second, run tests to check drive throughput and establish a baseline for future comparison
$ cd /root/post-install
$ clush -Ba '/root/post-install/runRWSpeedTest.sh' | tee runRWSpeedTest.log
2. Application Benchmarking
Use specific MapReduce jobs to create test data and process it in order to challenge the performance limits of the cluster.
First, create a volume for the test data
$ maprcli volume create -name benchmarks -replication 1 -mount 1 -path /benchmarks
Second, generate random sequence of data
$ su mapr
$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar teragen 5000000 /benchmarks/teragen1
Then, sort the data and write the output to a directory
$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar terasort /benchmarks/teragen1 /benchmarks/terasort1
To analyze how long it takes to perform each step check the logs on the JobHistoryServer
$ clush -a jps | grep -i JobHistoryServer
The 3 drives per pool gives us a good balance read/write speed for ingestion huge data and recovery time for failed drives.
Storage pools hold units called Containers (32Gb size by default) which are logically organized into Volumes (which are specific to MapR FS). By default, containers has replication factor inside a volume set to three. We can choose a pattern for replication across containers: chain pattern, star pattern.
$ maprcli volume create name type 0|1
When writing for a file, Container Location Database (CLDB) is used to determine first container where data is written. CLDB replaces the function of a NameNode in MapR hadoop, it stores container replication factor and pattern information. A file is divided into chuncks (default size 256 Mb): small chunk size leads to high writes scheduling overhead, big chunk size requires more memory.
A topology defines the physical layout of a cluster nodes. It's recommended to have two top-level topologies:
Snapshots are read-only images of volumes at a specific point in time, more accurately a pointer that costs almost nothing. It's good idea to create them regularly to protect the integrity of the data. By default, a snapshot is scheduled automatically at the creation of a volume, it can be customized through the MCS or manually created as follows:
$ maprcli volume snapshot create -volume DemoVolume -snapshotname 12042015-DemoSnapshot
Mirrors are volumes that represents an exact copy of a source volume from same or different cluster, it takes an extra amount of resources and time to create them. By default, a mirror is a read-only volume but can be made writable. They can be created through the MCS, set the replication factor or manually as follows:
$ maprcli volume mirror start -name DemoVolumeMirror
$ maprcli volume mirror push -name DemoVolumeMirror
Configuring remote mirrors
First, edit cluster configuration file (in both clusters) to include the location of CLDB nodes on the remote one:
$ echo "cldb_addr1:7222 cldb_addr2:7222 cldb_addr3:7222" >> /opt/mapr/conf/mapr-clusters.conf
Second, copy this new configuration to all nodes in the cluster
$ clush -a --copy /opt/mapr/conf/mapr-clusters.conf
Third, restart the Warden service so that the modification takes effect:
$ clush -a service mapr-warden restart
Finally, start the mirroring from the MCS interface.
Standard logs for each node are stored at /opt/mapr/hadoop/hadoop-2.5.1/logs, however the centralized logs are stored in /mapr/MaprQuickInstallDemo/var/mapr/local/c200-01/logs at the cluster level.
Centralized logging automate for us the gathering of logs from all cluster nodes. It provides a job-centric view. The following command can be used to create a centralized log direcotry populated with symbolic links to all log files related to: tasks, map attempts, reduce attempts, pretaited to this specific job.
$ maprcli job linklogs -jobid JOB_ID -todir MAPRFS_DIR
The MapR centralized logging feature is enabled by default in /opt/mapr/hadoop/hadoop-0.20.2/conf/hadoop-env.sh through the environement variable HADOOP_TASKTRACKER_ROOT_LOGGER.
Standard log for each node is stored under /opt/mapr/hadoop/hadoop-0.20.2/logs,
on the other hands the centralized logs are stored in the /map/ when starting at the cluster level.
Alarms
When a disk failure alarm is raised, the report at /opt/mapr/logs/faileddisk.log gives information about what disks have failed, the reason of the failure and recommended resolution.
Schedule Maintenance Jobs
The collected metrics give us a good view of the cluster performance and health. The complexity of the cluster makes it hard to use these metrics to optimize how the cluster is running.
Running test jobs regularly to gather job statistics and watch cluster performance. If a variance in the cluster performance can be seen the actions need to be taken to get back the cluster performance. By doing this in a controlled environment we can try different ways (e.g. tweak Disk and Role balancers settings) to optimize the cluster performance.
- Testing & verifying Hardware before installing MapR Hadoop
- Installing MapR Hadoop
- Benchmarking MapR Hadoop Cluster configure new cluster for production
- Monitoring Cluster for failures & performance
Prerequisites
Install the cluster shell utility, declare the slave nodes and check if it is accessing the nodes properly
$ sudo -i
$ apt-get install clustershell
$ apt-get install clustershell
$ mv /etc/clustershell/groups /etc/clustershell/groups.original
$ echo "all: 192.168.2.212 192.168.2.200" > /etc/clustershell/groups
$ echo "all: 192.168.2.212 192.168.2.200" > /etc/clustershell/groups
$ clush -a date
Mapr Cluster validation
Inconsistency in the hardware (e.g. different disk sizes or cpu cores) may not cause installation failure but may cause poor performance of the cluster. The use of benchmarking tools (cluster validation github repo) allows the measurement of the cluster performance.
The remaining of this section address pre-Install cluster hardware tests:
1. Download Benchmark Tools
2. Prepare Cluster Hardware for Parallel Execution of Tests
3. Test & Measure Subsystem Components
4. Validate Component SOftware & Firmware
Grap the validation tools from the github repo
$ curl -L -o cluster-validation.tgz http://github.com/jbenninghoff/cluster-validation/tarball/master
$ tar xvzf cluster-validation.tgz
$ mv jbenninghoff-cluster-validation-*/ ./
$ cd pre-install/
Copy the pre-install folder to all nodes, and check if it succeeded
$ clush -a --copy /root/pre-install/
$ clush -a ls /root/pre-install/
Test the network bandwidth for its ability to handle MapReduce operations:
First, set the IP addresses of the node in network-test.sh (divide them between half1 and half2).
$ /root/pre-install/network-test.sh | tee network-test.log
Test memory performance
$ clush -Ba '/root/pre-install/memory-test.sh | grep ^Triad' | tee memory-test.log
The remaining of this section address pre-Install cluster hardware tests:
1. Download Benchmark Tools
2. Prepare Cluster Hardware for Parallel Execution of Tests
3. Test & Measure Subsystem Components
4. Validate Component SOftware & Firmware
$ curl -L -o cluster-validation.tgz http://github.com/jbenninghoff/cluster-validation/tarball/master
$ tar xvzf cluster-validation.tgz
$ mv jbenninghoff-cluster-validation-*/ ./
$ cd pre-install/
Copy the pre-install folder to all nodes, and check if it succeeded
$ clush -a --copy /root/pre-install/
$ clush -a ls /root/pre-install/
Test the hardware for specification heterogeneity
$ /root/pre-install/cluster-audit.sh | tee cluster-audit.logTest the network bandwidth for its ability to handle MapReduce operations:
First, set the IP addresses of the node in network-test.sh (divide them between half1 and half2).
$ /root/pre-install/network-test.sh | tee network-test.log
Test memory performance
$ clush -Ba '/root/pre-install/memory-test.sh | grep ^Triad' | tee memory-test.log
Test disk performance
The disk-test.sh script checks the disk health and performance (i.e. throughput for sequential and random I/O read/write), it destroys any data available on it.
$ clush -ab /root/pre-install/disk-test.sh
For each scanned disk there will be a result file of the form disk_name-iozone.log.
Mapr Quick Install - link
Minimum requirements:- 2-4 cores (at least two: 1 CPU for OS, 1 CPU for filesystem)
- 6GB of ram
- 20GB size of raw disk (should not be formatted/partitioned)
First, download installer script
$ wget http://package.mapr.com/releases/v4.1.0/ubuntu/mapr-setup
$ chmod 755 mapr-setup
$ ./mapr-setup
Second, configure the installation process (e.g. define data and control nodes). A sample configuration can be found in /opt/mapr-installer/bin/config.example
$ cd /opt/mapr-installer/bin
$ cp config.example config.example.original
Use following commands to find information on nodes to declare in the configuration
$ clush -a lsblk # list drivers name
$ clush -a mount # list ip addresses and mounted drivers
Edit config.example file
- Declare the nodes information (IP addresses and data drives) under the Control_Nodes section.
- Customize the cluster domain by replacing my.cluster.com with your own.
- Set a new password (e.g. mapr)
- Declare the disks and set ForceFormat to true.
$ ./install --cfg config.example --private-key /root/.ssh/id_rsa -u root -s -U root --debug new
 |
| MapR Cluster Services - link |
Otherwise, if the installation fails, then remove install folder then check installation logs that can be found at /opt/mapr-installer/var/mapr-installer.log. Example of failures may be caused by:
- problems formatting disks for MapR FS (check /opt/mapr/logs/disksetup.0.log).
- one of the nodes has less than 4G of memory
- disks with LVM setup
$ rm -r -f /opt/mapr/ # remove installation folder
$ dpkg --get-selections | grep -v deinstall | grep mapr
mapr-cldb install
mapr-core install
mapr-core-internal install
mapr-fileserver install
mapr-hadoop-core install
mapr-hbase install
mapr-historyserver install
mapr-mapreduce1 install
mapr-mapreduce2 install
mapr-nfs install
mapr-nodemanager install
mapr-resourcemanager install
mapr-webserver install
mapr-zk-internal install
mapr-zookeeper install
$ dpkg -r --force-depends
To check if the cluster is running properly, we can run the following quick test job.
Note: check that the names of cluster nodes are resolvable through DNS, otherwise declare them in the /etc/hosts of each node.
$ su - mapr
$ cd /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
$ yarn jar hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi 8 800
Benchmark the Cluster
1. Hardware BenchmarkingFirst, copy the post-install folder to all nodes
$ clush -a --copy /root/post-install
$ clush -a ls /root/post-install
Second, run tests to check drive throughput and establish a baseline for future comparison
$ cd /root/post-install
$ clush -Ba '/root/post-install/runRWSpeedTest.sh' | tee runRWSpeedTest.log
2. Application Benchmarking
Use specific MapReduce jobs to create test data and process it in order to challenge the performance limits of the cluster.
First, create a volume for the test data
$ maprcli volume create -name benchmarks -replication 1 -mount 1 -path /benchmarks
Second, generate random sequence of data
$ su mapr
$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar teragen 5000000 /benchmarks/teragen1
Then, sort the data and write the output to a directory
$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar terasort /benchmarks/teragen1 /benchmarks/terasort1
To analyze how long it takes to perform each step check the logs on the JobHistoryServer
$ clush -a jps | grep -i JobHistoryServer
Cluster Storage Resources
MapR FS organizes the drives of a cluster into Storage Pools. The later is a group of drives (three by default) on a single physical node. Data is stored across drives of the cluster storage pools. In case, one drive fails then the entire storage pool is lost. To recover it, we need to put all drives of this pool offline, replace the failed drive then return then back to the cluster.The 3 drives per pool gives us a good balance read/write speed for ingestion huge data and recovery time for failed drives.
Storage pools hold units called Containers (32Gb size by default) which are logically organized into Volumes (which are specific to MapR FS). By default, containers has replication factor inside a volume set to three. We can choose a pattern for replication across containers: chain pattern, star pattern.
$ maprcli volume create name
When writing for a file, Container Location Database (CLDB) is used to determine first container where data is written. CLDB replaces the function of a NameNode in MapR hadoop, it stores container replication factor and pattern information. A file is divided into chuncks (default size 256 Mb): small chunk size leads to high writes scheduling overhead, big chunk size requires more memory.
A topology defines the physical layout of a cluster nodes. It's recommended to have two top-level topologies:
- /data the parent topology for active nodes in the cluster
- /decommissioned the parent topology used to segregate offline nodes or nodes to be repaired.
Data Ingestion
Ingestion data to MapR FS can be done through:- NFS (e.g. Gateway Strategy, Colocation Strategy) by using traditional applications with multiple concurrent read/writes easily - link,
- Sqoop to transfer data between MapR-FS and relational databases,
- Flume a distributed service for collecting, aggregating & moving data into MapR-FS
Snapshots are read-only images of volumes at a specific point in time, more accurately a pointer that costs almost nothing. It's good idea to create them regularly to protect the integrity of the data. By default, a snapshot is scheduled automatically at the creation of a volume, it can be customized through the MCS or manually created as follows:
$ maprcli volume snapshot create -volume DemoVolume -snapshotname 12042015-DemoSnapshot
Mirrors are volumes that represents an exact copy of a source volume from same or different cluster, it takes an extra amount of resources and time to create them. By default, a mirror is a read-only volume but can be made writable. They can be created through the MCS, set the replication factor or manually as follows:
$ maprcli volume mirror start -name DemoVolumeMirror
$ maprcli volume mirror push -name DemoVolumeMirror
Configuring remote mirrors
First, edit cluster configuration file (in both clusters) to include the location of CLDB nodes on the remote one:
$ echo "cldb_addr1:7222 cldb_addr2:7222 cldb_addr3:7222" >> /opt/mapr/conf/mapr-clusters.conf
Second, copy this new configuration to all nodes in the cluster
$ clush -a --copy /opt/mapr/conf/mapr-clusters.conf
Third, restart the Warden service so that the modification takes effect:
$ clush -a service mapr-warden restart
Finally, start the mirroring from the MCS interface.
Cluster Monitoring
Once a cluster is up and running, it has to be kept running smoothly. MCS provides many tools to monitor the health and to investigate failure causes of the cluster by providing:- alarms: sending emails, nagios notification, and
- statistics about nodes (e.g. services), volumes, jobs (MapR metrics database). MapR Hadoop provide ways to
Standard logs for each node are stored at /opt/mapr/hadoop/hadoop-2.5.1/logs, however the centralized logs are stored in /mapr/MaprQuickInstallDemo/var/mapr/local/c200-01/logs at the cluster level.
Centralized logging automate for us the gathering of logs from all cluster nodes. It provides a job-centric view. The following command can be used to create a centralized log direcotry populated with symbolic links to all log files related to: tasks, map attempts, reduce attempts, pretaited to this specific job.
$ maprcli job linklogs -jobid JOB_ID -todir MAPRFS_DIR
The MapR centralized logging feature is enabled by default in /opt/mapr/hadoop/hadoop-0.20.2/conf/hadoop-env.sh through the environement variable HADOOP_TASKTRACKER_ROOT_LOGGER.
Standard log for each node is stored under /opt/mapr/hadoop/hadoop-0.20.2/logs,
on the other hands the centralized logs are stored in the /map/ when starting at the cluster level.
Alarms
When a disk failure alarm is raised, the report at /opt/mapr/logs/faileddisk.log gives information about what disks have failed, the reason of the failure and recommended resolution.
Cluster Statistics
MapR collects a variety of statistics about the cluster and running jobs. There information helps track the cluster usage and health. They can be writting to an output file or consumed by ganglia, the output type is specified in two hadoop-metrics.properties files:- /opt/mapr/hadoop/hadoop-0.20.2/conf/hadoop-metrics.properties for output of hadoop standard services
- /opt/mapr/conf/hadoop-metrics.properties for output of MapR specific services
Schedule Maintenance Jobs
The collected metrics give us a good view of the cluster performance and health. The complexity of the cluster makes it hard to use these metrics to optimize how the cluster is running.
Running test jobs regularly to gather job statistics and watch cluster performance. If a variance in the cluster performance can be seen the actions need to be taken to get back the cluster performance. By doing this in a controlled environment we can try different ways (e.g. tweak Disk and Role balancers settings) to optimize the cluster performance.
Resources:
- MapR installation - Lab Guide, Quick Installation Guide
- Preparing Each Node - link
- Setting up a MapR Cluster on Amazon Elastic MapReduce - link
- Cluster service planning - link
- Tuning cluster for MapReduce performance for specific jobs - link
- MapR Hadoop data storage - link
