Configuration Management Tools

 

Building an base image (Amazon Machine Image (AMI)) using Packer

 Requirement:

Create and upload a new ubuntu image (AMI) in amazon cloud using packer and version control.

Steps Involved: 

1. Create a free-tier account with aws.amazon.com
2. Install packer in your local system
3. Configure the aws user account to authenticate from packer
4. Define and Create the base config for building a base image using package HCP format
5. Run the packer utility and pass the config file.
6. Validate the first base image in was management console.

How to install brew command-line utility in macOS

Requirement:

In order to install packages in macOS , we use "brew" command-line utility.

Installation Steps:

For installing "brew" utility follow below steps:

1. (As a normal user) Download the install.sh script
• curl -O https://raw.githubusercontent.com/Homebrew/install/master/install.sh
2. (Provide execute permission to the downloaded script file install.sh)
• chmod +x ./install.sh
3. (Start the script execution)
• ./install.sh
4. (Add the default Homebrew install path to default .zprofile)

• echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/$USER/.zprofile

• eval "$(/opt/homebrew/bin/brew shellenv)"

Finally execute the command brew help and test the installation

How to make a user part of group in macOS Big Sur 11.6

Scenario: You are using the MacBook (with macOS Big Sur 11.6) for the first time and you wanted to install "brew" (or) "wget" (or) any other command-line utility

Let's consider user name is "webuser" and you wanted to make this user part of a "wheel" group to execute sudo commands. Use the command "dseditgroup"

$ (Run the command as root user in macOS)

$ dseditgroup -o edit -a webuser -t user wheel

o: Operation - read, create, edit, delete , checknumber 

a: Name of the record to be added to the group specified by group name

t: Type of record to be added to or deleted from group specified by groupname. Valid values are user, computer , group or computergroup

wheel: groupname

DevOps | Automation | Migration :: Subversion (SVN) to GitHub Enterprise Migration with 500GB+ data and 50,000+ Commits.

 

Subversion is a Centralized Version Control system and GitHub Enterprise is distributed version control system.

In one of the project I recently lead the team for migrating code-base with over 10+yrs old application code-base of 500+GB data of telecom software application, including commit history from legacy SVN to Enterprise grade GitHub. Supported in validating the workflow with muti-project dependencies for over 46+ components successfully.Some of features used are described briefly below:-

SVN:

1. SVN Externals was handling multi-module project dependencies in Subversion.
2. All the artifacts of the application code base were directly stored in Subversion server.

GitHub Enterprise:

1. GitHub was used to version control code/config or human readable file format. The maximum file size it supported was 100MB as de facto standard.
   
2. Binaries/Artifacts were version controlled using GitLFS (Long File Storage)
3. Multi-project dependencies were handled by Git Sub-module

Bridging of SVN and GitHub:

1. git-svn utility was used to convert the SVN revision no based commits to GitHub Commit SHA-IDs
2. Shell Script was used to handle preparation of authors file , import of SVN data and upload to GitHub
3. GitHub API integration with Terraform were used for handling org/repo/team/GitHub branch protection.
4. "git svn rebase" operation can be used to migrate the delta changes over a period of time.
5. Jenkins Automation Server configuration to migrate SVN jobs to GitHub.

Developer Validation:

1. sourcetree from Atlassian is best suited tool for - integrating GitHub/GitLFS/Git Submodule as a one stop solution.
2. Sourcetree tool supports GitHub , GitHub Enterprise , GitLab and Bit Bucket
   

I hope the post will be useful for folks working in DevOps and Managing the software version control using SVN and if you have any plan in pipeline to migrate to GitHub Enterprise.

Challenges:

1. Migration from SVN to GitHub was not free from challenges, we had quite a bunch in managing SVN index data generated from git-svn utility.
2. We had commit history ranging from 100 to 50,000+ commits and it took days to migrate a specific telecom product releases.
3. git-filter branch utility was used to clear index data reference, matching false positive files which blocked the migration or push operation to GitHub Enterprise repositories. 

 

I would like to hear your experience if you have been going through any issues in your SVN migration activity  and love to support if you have face any challenges.

 

References:

1. https://www.sourcetreeapp.com/  - Free Sourcetree tool
2. https://www.atlassian.com/git/tutorials/migrating-convert - git-svn utility
3. https://www.hashicorp.com/blog/managing-github-with-terraform - GitHub API and Terraform onboarding
   

 

Happy-new-year-2020

Real happiness is in ..

1. unconditional love
2. sharing knowledge 
3. Forget & Forgive
4. Be listenable 
5. winning yourself
6. smile  at others
7. daily exercise
8. Involve in sports
9. Play with kids
10. Talk over texting
11. Live the moment
12. Failing and rising  
13. brush twice a day
14. take a family  medical insurance
15. Take a term policy
16. Involve in your passion or hobby daily
17. Improve on a second profession
18. Music On - Dance alone 
19. listen to  light music
20. Go office  early 😀 and reach home safe (wear seat belt and helmet)
21. Happy new year 2020 wishes to all my friends and family members

How to remap/restore the openstack volume data

How to remap/restore the openstack volume data, post rebuilding the instance (by preserving/restoring its old IP address):

Issue: 

If an instance went to non-recoverable state, due to back-end hypervisor hardware (or) software failure, application team are forced to create a new instance from the base image by restoring its IP address and volume data to bring the application back online.

Message/logs/symptoms of Hypervisor Failure:

[Errno 30] Read-only file system: '/var/lib/nova/instances/.directio.test'

Name

prod-app1-os-chn-web-1

ID

ddd878bd-a3434-333-a3fb-a45r456yy432

Status

Error

Availability Zone

nova

Created

Jul. 07, 2018, 6:01 A.m.

Time Since Created

10 months

Fault

• Instance is in an unrecoverable state.

• Instance was not accessible. Soft/Hard reboot did not fix, and now instance is in a "No State" status.
• Instance hosted on oscomp-chn-h01 which is currently down due to disk failure. Disk replacement required which results to data loss. Openstack team recommends to destroy and rebuild the instance.

Importance of Preserving the IP of an cloud instance:-

• Any firewall changes by firewall team done for enabling the infra  access (such as Database) with this instance IP, will not have to be repeated. 
• Any front-end server (haproxy) which is mapped to this backend instance IP (in haproxy.cfg), don't have to be changed.
• Any changes to GSLB (Global Server Load Balancing) for mapping the global domain name (eg: redhat.com) backend server ip address, need not be changed.

Background of Openstack:

Openstack is a public/private cloud platform  for hosting the virtual machines in an hypervisor, for an application to be hosted in cloud, with the below list of essential cloud infrastructure services and its purpose.

• Keystone - Authentication/Authorization
• Neutron - Networking
• Glance - Instance Images
• Swift -  Unstructured Storage / Object BLOB (Binary Large Object)
• Cinder - Block Storage
• Heat - Openstack Orchestration 
• Horizon - Openstack Dashboard
• Ceilometer - Monitoring / Metering
• Nova - Computing (cpu/memory/disk)
  
  

If you would like to take a look at the additional services of openstack , refer to openstack-components

Steps to restore openstack volume data (once the instance is back online after preserving the IP address):

Assuming the application team restored the instance and mapped the openstack volume data of the instance under disk name /dev/vdb.

Our aim is to restore the 500GB volume data (in the original server path /var/logs/webserver_logs ) and bring back the application online.

In the Linux instance explained in this scenario, the data in the original server path /var/logs/webserver_logs is organized in terms of physical volume / volume group / logical volumes / application mount point/folder name/path. 

In our case 500GB volume data is stored in terms of:

• physical volume name: /dev/vdb
• volume group name: vg_web_logs
• logical volume name: lv01_web_logs
• application mount point/folder name/path in linux : /var/logs/webserver_logs

1. Makes sure the newly re-built (centOS 7.x is the operating system image) instance has the lvm2 (Logical Volume Manager) package installed.

      2. lvmdiskscan command : Scans all SCSI, (E)IDE disks, multiple devices and a bunch of other block devices in the system looking for LVM physical volumes.

 3. lvscan command : Scans all known volume groups or all supported LVM block devices in the system for defined Logical Volumes. The output consists of one line for each Logical Volume indicating whether or not it is active, a snapshot or origin, the size of the device and its allocation policy. 

4. modprobe command : To trigger the kernel module. Scan your system for LVM volumes and identify in the output the volume group name that has your Centos 7.x volume (in this case volume group name from old instance is vg_app1_logs):

5. vgchange command: Sets the availability of a volume group to yes or no

6. Issue lvscan command again to activate the logical volume

7.  The Device-mapper is a component of the Linux kernel (since version 2.6) that supports logical volume management. It is required by LVM2 and EVMS Enterprise Volume Management System. The original LVM (included in stock 2.4 kernels) does not use it.If you intend to use Device-mapper, as well as including it in your kernel, you should install the userspace configuration tool (dmsetup) and library (libdevmapper).

8. And finally mount the volume data from the logical volume device path /dev/vg_web_logs/lv01_web_logs to file system mount point/path /var/logs/webserver_logs ,we have restored in the application volume  path /var/logs/webserver_logs a data of 22GB data present out of 500GB disk space.

9. df command: disk free command output below shows the restored volume data like Filesystem | Type | Size | Used | Available Space | Use % and Mount Point path in server/instance.

10. Congratulations!. Now we can go ahead and bring the application back online, in this case it is a web server.

References:

1. https://www.openstack.org/software/project-navigator/openstack-components#openstack-services
2. https://linux.die.net/man/8/