Elasticity Using LVM (Logical Volume Management)

Below ScreenShots will give u detailed information to configure namenode

  1. Check whether jdk and hadoop package installed on namenode using command

Below ScreenShots will give u detailed information to configure datanode

  1. Check whether jdk and hadoop package installed on namenode using command

Starting Connectivity

On Namenode Side

  1. Use hadoop-daemon.sh start namenode command to start the namenode
  2. Use jps command to check namenode started or not
  3. Use netstat -tnlp to check whether hadoop server started or not
  1. Use hadoop-daemon.sh start datanode command to start the datanode
  2. Use jps command to check namenode started or not
  1. Only one datanode connected

STEP 1 :-Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

Now increasing datanode storage using lvm

On Datanode Side

  1. Use fdisk -l command to list all the hard disks attached to this vm.
  1. Convert the virtual hard disk volume to physical volume using command pvcreate disk_name

Setting Connectivity

On Namenode Side

  1. Use hadoop-daemon.sh stop namenode command to stop the namenode
  2. Use hadoop-daemon.sh start namenode command to again start the namenode
  3. Use jps command to check namenode started or not
  4. Use netstat -tnlp to check whether hadoop server started or not
  1. Use hadoop-daemon.sh stop datanode command to stop the datanode
  2. Use jps command to check datanode stoped or not.In my case my datanode stoped.
  3. Use hadoop-daemon.sh start datanode command to again start the datanode
  4. Use jps command to check datanode started or not
  1. Only one datanode connected

STEP 2 :- Increase or Decrease the Size of Static Partition in Linux.

Extending lvm

On Datanode Side

  1. First datanode is contributing /lvm1 space (4GB can say 3.64 GB ) to namenode . Now we want to extend the space contributed by lvm1.
  2. To extend/increase the size of lvm1 use command lvextend — — size +increase_size /dev/vg_name/lv_name

Setting Connectivity

On Namenode Side

  1. Use hadoop-daemon.sh stop namenode command to stop the namenode
  2. Use hadoop-daemon.sh start namenode command to again start the namenode
  3. Use jps command to check namenode started or not
  4. Use netstat -tnlp to check whether hadoop server started or not
  1. Use hadoop-daemon.sh stop datanode command to stop the datanode
  2. Use jps command to check datanode stoped or not.In my case my datanode stoped.
  3. Use hadoop-daemon.sh start datanode command to again start the datanode
  4. Use jps command to check datanode started or not
  1. Only one datanode connected

STEP 3 :- Automating LVM Partition using Python-Script.

This is an python-script code to automate LVM

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store