How to remap/restore the openstack volume data, post rebuilding the instance (by preserving/restoring its old IP address):
Issue:
If an instance went to non-recoverable state, due to back-end hypervisor hardware (or) software failure, application team are forced to create a new instance from the base image by restoring its IP address and volume data to bring the application back online.
Message/logs/symptoms of Hypervisor Failure:
[Errno 30] Read-only file system: '/var/lib/nova/instances/.directio.test'
Name
prod-app1-os-chn-web-1
ID
ddd878bd-a3434-333-a3fb-a45r456yy432
Status
Error
Availability Zone
nova
Created
Jul. 07, 2018, 6:01 A.m.
Time Since Created
10 months
Fault
- Instance is in an unrecoverable state.
- Instance was not accessible. Soft/Hard reboot did not fix, and now instance is in a "No State" status.
- Instance hosted on oscomp-chn-h01 which is currently down due to disk failure. Disk replacement required which results to data loss. Openstack team recommends to destroy and rebuild the instance.
Importance of Preserving the IP of an cloud instance:-
- Any firewall changes by firewall team done for enabling the infra access (such as Database) with this instance IP, will not have to be repeated.
- Any front-end server (haproxy) which is mapped to this backend instance IP (in haproxy.cfg), don't have to be changed.
- Any changes to GSLB (Global Server Load Balancing) for mapping the global domain name (eg: redhat.com) backend server ip address, need not be changed.
Background of Openstack:
Openstack is a public/private cloud platform for hosting the virtual machines in an hypervisor, for an application to be hosted in cloud, with the below list of essential cloud infrastructure services and its purpose.
- Keystone - Authentication/Authorization
- Neutron - Networking
- Glance - Instance Images
- Swift - Unstructured Storage / Object BLOB (Binary Large Object)
- Cinder - Block Storage
- Heat - Openstack Orchestration
- Horizon - Openstack Dashboard
- Ceilometer - Monitoring / Metering
- Nova - Computing (cpu/memory/disk)
If you would like to take a look at the additional services of openstack , refer to openstack-components
Steps to restore openstack volume data (once the instance is back online after preserving the IP address):
Assuming the application team restored the instance and mapped the openstack volume data of the instance under disk name /dev/vdb.
Our aim is to restore the 500GB volume data (in the original server path /var/logs/webserver_logs ) and bring back the application online.
In the Linux instance explained in this scenario, the data in the original server path /var/logs/webserver_logs is organized in terms of physical volume / volume group / logical volumes / application mount point/folder name/path.
In our case 500GB volume data is stored in terms of:
- physical volume name: /dev/vdb
- volume group name: vg_web_logs
- logical volume name: lv01_web_logs
- application mount point/folder name/path in linux : /var/logs/webserver_logs
- Makes sure the newly re-built (centOS 7.x is the operating system image) instance has the lvm2 (Logical Volume Manager) package installed.
2. lvmdiskscan command : Scans all SCSI, (E)IDE disks, multiple devices and a bunch of other block devices in the system looking for LVM physical volumes.
3. lvscan command : Scans all known volume groups or all supported LVM block devices in the system for defined Logical Volumes. The output consists of one line for each Logical Volume indicating whether or not it is active, a snapshot or origin, the size of the device and its allocation policy.
4. modprobe command : To trigger the kernel module. Scan your system for LVM volumes and identify in the output the volume group name that has your Centos 7.x volume (in this case volume group name from old instance is vg_app1_logs):
5. vgchange command: Sets the availability of a volume group to yes or no
6. Issue lvscan command again to activate the logical volume
7. The Device-mapper is a component of the Linux kernel (since version 2.6) that supports logical volume management. It is required by LVM2 and EVMS Enterprise Volume Management System. The original LVM (included in stock 2.4 kernels) does not use it.If you intend to use Device-mapper, as well as including it in your kernel, you should install the userspace configuration tool (dmsetup) and library (libdevmapper).
8. And finally mount the volume data from the logical volume device path /dev/vg_web_logs/lv01_web_logs to file system mount point/path /var/logs/webserver_logs ,we have restored in the application volume path /var/logs/webserver_logs a data of 22GB data present out of 500GB disk space.
9. df command: disk free command output below shows the restored volume data like Filesystem | Type | Size | Used | Available Space | Use % and Mount Point path in server/instance.
10. Congratulations!. Now we can go ahead and bring the application back online, in this case it is a web server.
References:
- https://www.openstack.org/software/project-navigator/openstack-components#openstack-services
- https://linux.die.net/man/8/
