Begin with Terraform

Completely different 😀 !!

In my recent project I have been involved more into AWS technologies where I need to connect java application with AWS and majorly into infrastructure part but not using AWS console but using TERRAFORM.

What I will be covering in this blog:

• What is terraform?
• Prerequisite - How to connect with AWS?
• Environment information 
• Versioning
• Scripts creating
• AWS Config
• User(s)
• Role(s)
• User-Role relationship
• Group(s)
• Get latest available AMI of linux in your region
• Create EC2 instance using that AMI - and connect using ssh
• Print useful information
• Codebase
• Beginner
• Update 2 - Packer and Modules
• Update 3 - ASG, SSL, ELB, User-data
• Commands
• Glossary
• Images to refer

What is terraform?
Visit terraform-website for details.  But in short it is an automation tool to different cloud providers for managing infrastructure.  

Example - If someone asks you to create 500 EC2 instances or 100 ASGs (Auto Scaling Group), then you need something like terraform.  Make sense?

BTW - I am using terraform version 0.12 for this use case.


Prerequisite - How to connect with AWS?
There are some initial things that you need to do.

• Obviously terraform must be installed
• You must have AWS account - I am using free tier
• For security reasons it is best to have admin group in IAM

Environment information 

• In this use case all terraform (.tf) files are in one single folder.
• All .tf files will be executed/processed when you run any terraform command.
• AWS-CLI and create profile for a user - create config and credentials
• Two files config and credentials must be present under .aws folder of a USER_HOME.
• Terraform picks aws_access_key_id and aws_secret_access_key from credentials file only, so NO need to feed secure information in terraform script.
• Use case carried on OS Win 10 using cmd prompt.

Versioning
Version of software used:

• OS: Win10 Home
• Terraform: v0.12.18
• Packer: 1.5.1
• AWS: aws-cli/1.16.303 Python/3.6.0 Windows/10 botocore/1.13.39

How to set
Let's see various settings.

• AWS Config
• One common file is created which will be used to validate user for all functions/tf-files.
• Refer file aws-config.tf
• User(s)
• I read multiple documents and it is recommended to create-user via console, so I created in same way.  Still refer file create-user-UNUSED.tf
• Group(s)
• User created must belong to group, so whatever group can do that user can do.
• Refer file create-group.tf
• Role(s)
• Roles created and can be assigned to a user.  Each role has its own policy defines which actions are allowed on which resource.
• A user can assume role based on his permissions to perform certain operation. 
• Refer files create-role.tf, create-role-policy.tf and dev-assume-role-policy.json
• User-Role relationship
• Relationship that defines which user has which role.
• Refer file create-user-membership.tf
• Get latest available AMI of linux in your region
• This is important one.  Each region has multiple AMIs.  I created one query which selects latest linux ami created by amazon.  So, file has that logic which connects to AWS and fetches id of AMI and EC2 uses that AMI to create an instance. 
• Refer file get-ami.tf
• Create EC2 instance using that AMI - and connect using ssh
• This file has multiple parts
• First it has logic to create instance from AMI selected in previous step.
• Creates key-pair (you need to create public-private ssh key, so we can login to instance via ssh)
• It passes public key file so that key will be associated to instance and using private key ssh operation should be successful.
• Do not worry if fingerprint generated by your command and AWS is different because logic used by AWS is different, so your private key will still work.
• Important - One last thing that took lot of time for me for ssh - you need to include your specific IP in inbound/outbound rule on all traffic, even though rule of 0.0.0.0/0 is still present.
• I will further analyse it.
• Update (Important) - Now this is no more required.  After creating public subnet and associate subnet with EC2, I can ssh instance using public IP without any other rule.  Cheers 👍 !! 
• Default user/password for ssh is ec2-user/<EMPTY> and you need to pass private-key.  Use public IP to of instance.
• Refer file create-instance.tf
• Print useful information
• This files tells logic of printing any values in case debugging is required.  Only this way I come to know that I misconfigured my region and it was selecting wrong ami.
• Refer file get-user-region.tf

Codebase

• Complete codebase
• https://github.com/svermaji/terraform/
• Beginner
• I would be creating multiple folders in near future for advance scripting, this one is for beginner
• https://github.com/svermaji/terraform/tree/master/1-beginner
• Update 2 - Packer and Modules
• For packer commands are packer validate FILE_NAME and packer build FILE_NAME 
• https://github.com/svermaji/terraform/tree/master/2-modules

• Update 3 - ASG, SSL, ELB, User-data
• Third update has code that will create ASG, and one ELB.  Connects ELB with ASG and EC2 instances with ELB.  Now you can access url via ELB.  
• For HTTPS (SSL), I created my own certificate so if you do same the in browser you might see warning. As per example ELB is listening on 3 ports - 80 (http), 443 (https) and 8000 (http)
• Example also tells about user-data that executes on EC2 startup. 
• https://github.com/svermaji/terraform/tree/master/3-asg
• Update 4 - is about KMS
• Update 5 - is about separting modules
• Update 6 - is about httpd logs configuration
• Update 7 - is about S3 and uploading httpd logs

Commands
Commands used:

• init (only one time - execute in folder where your codebase is present)

Repeatedly 

• plan
• refresh (if you want to print output values)
• apply
• destroy

Commands referred like init, plan, apply destroy etc has below exact commands used:

• To initialize terraform first time
• terraform init
• To create plan that will be applied later
• terraform plan -no-color -refresh=true -out=infra.tfplan
• To apply plan 
• terraform apply -refresh=true -auto-approve "infra.tfplan"
• To refresh, check changes and output values
• terraform refresh
• To destroy resources created by terraform.  
• terraform destroy -auto-approve

• -auto-approve: This flag bypasses manual intervention.
• -no-color: This flag prints all output in single standard color
• Whatever plan is generated should be used in apply command, in above case its infra.tfplan

Glossary

• EC2 - Elastic Compute Cloud
• ASG - Auto Scaling Group
• IAM - Integrated Access Management
• AMI - Amazon Machine Image

Images to refer

Image-1: Result of apply command

Image-2: AWS Instance

Image-3: SSH to an instance

Image-4: Result of destroy command

Encryption on UI and Decryption on Server using Spring with AES 128 bit algorithm

Hi

Problem taken
Perform encryption of sensitive data at client/UI side using java script and perform decryption at Server end.


This is again debatable but most of enterprise applications; specially Banking applications; demands client side encryption for their sensitive data.

I had a discussion with some technical personalities in my current company as well and reached to conclusion that applying encryption at UI or Javascript side is really a very weak choice.  
Reason - To perform encryption at client side that means on UI via Javascript, Server needs to send secret key at client side and this is the place where integrity can be compromised.  There is no other way.

Still I am writing this blog where I personally made some tweaks to make algorithm little bit random in nature so that it takes more time for some one to crack.


Technology stack

• Spring 4.x
• CryptoJS (https://code.google.com/p/crypto-js/)
• Javascript

Proposed Solution

With the use of Crypto JS Interoperable encryption library certain fields which carry sensitive information will be encrypted by the system (on UI/Server) and will be decrypted at another system (Server/UI).  Once decryption performed successfully, process will be carry forwarded; and in case of failure request will be discarded.

In creating encrypted value for a data there are few parameters that are required by Crypto JS library which are:

·        secret-password

·        iv

·        salt 

All these three variables will be generated randomly for each encryption request.  System time will be taken into account along with HTML encryption to ensure uniqueness.  

Ex of encrypted data

“Text to encrypt” 

            encrypted to 

“9CP3Mi6pwLVyMBmLAssZS2gqQw4O4KXV6qfnB5S%2ByoY%3D”  

Complete Flow

UI Side

1. Time stamp is generated
2. A Random number is generated
3. Time stamp multipled with random number
4. Then above data converted to hexadecimal
5. Then same data is converted to hexadecimal conversion as per ascii table for non-numeric characters, ex. '%65' for 'A'
6. This encrypted string will be used as key (pass-phrase)
7. IV will be generated randomly
8. Salt will be generated randomly
9. Cipher Text will created using IV, Salt and Key
10. Now we have 6 parameters (cipher-text, iv, salt, pass-phrase, iteration-count (constant), keysize (constant))
11. These 6 parameters will be added as a string but every time at random position and there positions will be given to server as well.
1. Positions will be generated in numeric way as below
2. Ex. 043512 or 120354 or 403512 …
3. Now out of these 6 digits we will pick any 3 digits at random and convert into corresponding char like ‘a’ for 0, ‘b’ for 1 etc
4. Ex 043512  -> 0ed5b2
12. This position string will be appended to encryption string and send to server

Server Side

1. Server will split values based on delimiters (COMMA)
2. Then take final variable which will be representing there indices
3. Character will be converted to number
4. Final parameters will be retrieved
5. CipherText will be decrypted using "AES/CBC/PKCS5Padding".

Ex. of a encrypted (Green is separator and red are positions) text ce8ebf95d268caa811830afa922d92dc__bcdef567kop48__2227026d7ef9a18dae6394332378d40c__bcdef567kop48__128__bcdef567kop48__1000__bcdef567kop48__/+PdrEhznCcTEb/r+vlilA==__bcdef567kop48__%31%34%66%65%65%65%66%36%33%31%301vjqgqutx8fyb1ulwsx3s5ef74__bcdef567kop48__1,c,5,d,4,a

Complete code with sample is present at my GitHub.

Thanks

Shailendra

Spring boot and @RepositoryRestResource example with Hibernate and no Controller/Service layer and HATEOS

Hi,

Spring boot part two.  

Problem taken

In this blog I am integrating Spring boot with Hibernate.  In this I will demonstrate how easy it is to do following things:

• Connect with Hibernate
• Create Entity manager or datasource
• No need of Controller/Service layer
• Directly expose Repository as Rest path and access entity
• How to check entity links feature
• With only 3 to 4 files :)

Technology stack

• Java8
• Spring boot 1.3.x
• Spring 4.2.3
• Hibernate 4.x
• Apache Maven 3.3.x
• IDE - IntelliJ 14.x Community Edition

Solution

Just write few lines of code and you are done in generating Json.  How?  Let's see.

I just added pom.xml, one controller, one model, one repository and Application.java which will be starting point for Spring boot.  Since explanation wise there is not much I am directly coming to the code.

File pom.xml

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>spring</groupId>
    <artifactId>boot</artifactId>
    <version>1.0</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.0.RELEASE</version>
    </parent>

    <properties>
        <java.version>1.8</java.version>
        <start-class>com.spring.experiments.Application</start-class>
        <hibernate.version>4.2.11.Final</hibernate.version>
        <mysql.connector.version>5.1.36</mysql.connector.version>

        <!-- Logging -->        <logback.version>1.0.13</logback.version>
        <slf4j.version>1.7.5</slf4j.version>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
            <version>4.2.3.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-entitymanager</artifactId>
            <version>${hibernate.version}</version>
        </dependency>
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-jpa</artifactId>
            <version>1.9.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-rest-repository</artifactId>
            <version>1.0.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.connector.version}</version>
        </dependency>
        <dependency>
            <groupId>javax.inject</groupId>
            <artifactId>javax.inject</artifactId>
            <version>1</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-rest-webmvc</artifactId>
            <version>2.4.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-rest-core</artifactId>
            <version>2.4.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <version>1.4.182</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

File NotebookRepository.java

package com.spring.experiments.repository;

import com.spring.experiments.model.Notebook;
import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.data.rest.core.annotation.RepositoryRestResource;
import org.springframework.data.rest.repository.annotation.RestResource;

import java.util.List;

/** * Created by shaiverm on 23-Nov-2015 */@RepositoryRestResource(collectionResourceRel = "notebook", path = "notebook")
public interface NotebookRepository extends PagingAndSortingRepository<Notebook, Long> {

    List<Notebook> findByName(@Param("name") String name);

}

File Notebook.java

package com.spring.experiments.model;

import javax.persistence.*;
import java.sql.Date;

/** * Created by shaiverm on 17-Nov-2015 */@Entity@Table (name = "notebook")
public class Notebook {

    @Id    @Column(name="id")
    @GeneratedValue(strategy= GenerationType.IDENTITY)
    private Long id;

    private String name, description, tags, author;

    /*@Column(name = "created_on")    private Date createdOn;*/
    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getDescription() {
        return description;
    }

    public void setDescription(String description) {
        this.description = description;
    }

    public String getTags() {
        return tags;
    }

    public void setTags(String tags) {
        this.tags = tags;
    }

    public String getAuthor() {
        return author;
    }

    public void setAuthor(String author) {
        this.author = author;
    }

    /*public Date getCreatedOn() {        return createdOn;    }
    public void setCreatedOn(Date createdOn) {        this.createdOn = createdOn;    }*/
    @Override    public String
    toString() {
        return "Notebook{" +
                "id=" + id +
                ", name='" + name + '\'' +
                ", description='" + description + '\'' +
                ", tags='" + tags + '\'' +
                ", author='" + author + '\'' +
                //", createdOn=" + createdOn +                '}';
    }
}

File Application.java
package com.spring.experiments;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;

import java.util.Arrays;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        System.out.println("Spring Boot begins...");
        ApplicationContext ctx = SpringApplication.run(Application.class, args);

        System.out.println("Let's inspect the beans provided by Spring Boot:");

        String[] beanNames = ctx.getBeanDefinitionNames();
        Arrays.sort(beanNames);
        for (String beanName : beanNames) {
            System.out.println(beanName);
        }
        System.out.println("Spring Boot started...");
    }

}

File application.properties

spring.datasource.url=jdbc:mysql://localhost/testspring.datasource.username=rootspring.datasource.password=rootspring.datasource.driver-class-name=com.mysql.jdbc.Driver

Output Sample (Observe links with _links)
Url (Searching a record) - http://localhost:8080/notebook/search/findByName?name=Java8
{

• _embedded:
  

  {
  

  • notebook:
    

    [
    

    • {
      

      • name: "Java8",
      • description: "Book on features of Java8",
      • tags: "Java",
      • author: "Shailendra Verma",
      • _links:
        

        {
        

        • self:
          

          {
          

          • href: "http://localhost:8080/notebook/1"

          },
        • notebook:
          

          {
          

          • href: "http://localhost:8080/notebook/1"

          }

        }

      }

    ]

  },
• _links:
  

  {
  

  • self:
    

    {
    

    • href: "http://localhost:8080/notebook/search/findByName?name=Java8"

    }

  }

}

Url (List of records) - http://localhost:8080/notebook/
{

• _embedded:
  

  {
  

  • notebook:
    

    [
    

    • {
      

      • name: "Java8",
      • description: "Book on features of Java8",
      • tags: "Java",
      • author: "Shailendra Verma",
      • _links:
        

        {
        

        • self:
          

          {
          

          • href: "http://localhost:8080/notebook/1"

          },
        • notebook:
          

          {
          

          • href: "http://localhost:8080/notebook/1"

          }

        }

      },
    • {
      

      • name: "Health",
      • description: "Improve Health",
      • tags: "Nutrition",
      • author: "Doctor",
      • _links:
        

        {
        

        • self:
          

          {
          

          • href: "http://localhost:8080/notebook/2"

          },
        • notebook:
          

          {
          

          • href: "http://localhost:8080/notebook/2"

          }

        }

      }

    ]

  },
• _links:
  

  {
  

  • self:
    

    {
    

    • href: "http://localhost:8080/notebook"

    },
  • profile:
    

    {
    

    • href: "http://localhost:8080/profile/notebook"

    },
  • search:
    

    {
    

    • href: "http://localhost:8080/notebook/search"

    }

  },
• page:
  

  {
  

  • size: 20,
  • totalElements: 2,
  • totalPages: 1,
  • number: 0

  }

}


Url (Check entity details) - http://localhost:8080/profile/notebook
{

• alps:
  

  {
  

  • version: "1.0",
  • descriptors:
    

    [
    

    • {
      

      • id: "notebook-representation",
      • href: "http://localhost:8080/profile/notebook",
      • descriptors:
        

        [
        

        • {
          

          • name: "name",
          • type: "SEMANTIC"

          },
        • {
          

          • name: "description",
          • type: "SEMANTIC"

          },
        • {
          

          • name: "tags",
          • type: "SEMANTIC"

          },
        • {
          

          • name: "author",
          • type: "SEMANTIC"

          }

        ]

      },
    • {
      

      • id: "get-notebook",
      • name: "notebook",
      • type: "SAFE",
      • rt: "#notebook-representation",
      • descriptors:
        

        [
        

        • {
          

          • name: "page",
          • doc:
            

            {
            

            • value: "The page to return.",
            • format: "TEXT"

            },
          • type: "SEMANTIC"

          },
        • {
          

          • name: "size",
          • doc:
            

            {
            

            • value: "The size of the page to return.",
            • format: "TEXT"

            },
          • type: "SEMANTIC"

          },
        • {
          

          • name: "sort",
          • doc:
            

            {
            

            • value: "The sorting criteria to use to calculate the content of the page.",
            • format: "TEXT"

            },
          • type: "SEMANTIC"

          }

        ]

      },
    • {
      

      • id: "create-notebook",
      • name: "notebook",
      • type: "UNSAFE",
      • rt: "#notebook-representation"

      },
    • {
      

      • id: "delete-notebook",
      • name: "notebook",
      • type: "IDEMPOTENT",
      • rt: "#notebook-representation"

      },
    • {
      

      • id: "patch-notebook",
      • name: "notebook",
      • type: "UNSAFE",
      • rt: "#notebook-representation"

      },
    • {
      

      • id: "get-notebook",
      • name: "notebook",
      • type: "SAFE",
      • rt: "#notebook-representation"

      },
    • {
      

      • id: "update-notebook",
      • name: "notebook",
      • type: "IDEMPOTENT",
      • rt: "#notebook-representation"

      },
    • {
      

      • name: "findByName",
      • type: "SAFE",
      • descriptors:
        

        [
        

        • {
          

          • name: "name",
          • type: "SEMANTIC"

          }

        ]

      }

    ]

  }

}

How to Run

• mvn spring-boot:run

              or

• Use IDE by running Application.java

              or

• Make this application as webapp and deploy in Tomcat/Jetty or any web server.

Code at Github
I have posted my same code at Github.  Please visit below link to access it.

• GitHub

You can reach me for any queries at my email address.

Thanks

Shailendra