AWS CloudFormation: AutoScaling Group – You must use a valid fully-formed launch template

I had a AWS CloudFormation template that included an Auto Scaling Group, Launch Template and Security Group.

The Auto Scaling Group being created was to use the Launch Template being created. And that Launch Template was to use the Security Group being created.

When I created my CloudFormation stack, the stack failed with this error when my Auto Scaling Group was been created:

CREATE_FAILED You must use a valid fully-formed launch template. The parameter groupName cannot be used with the parameter subnet (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: e3c2b7e1-d94b-4a8d-a044-c15ba8791b62)

Even though the error says the issue is with the Auto Scaling Group, my issue was with my Launch Template. I attempted to specify the associated Security Group using the SecurityGroups property.

However, I needed to use the SecurityGroupIds property since I was not using the default VPC. The SecurityGroups property documentation states:

Security Groups

[EC2-Classic, default VPC] One or more security group names. For a nondefault VPC, you must use security group IDs instead. You cannot specify both a security group ID and security name in the same request.

Source: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-launchtemplate-launchtemplatedata.html#cfn-ec2-launchtemplate-launchtemplatedata-securitygroups

When I switched to using SecurityGroupIds, my stack would create.

Here is the CloudFormation template for my Launch Template and Auto Scaling Group:

EC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security Group for EC2 instances.
      #Other properties including SecurityGroupIngress, SecurityGroupEgress, VpcId
DemoLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateName: demo-launch-template
      LaunchTemplateData: 
        BlockDeviceMappings: 
          - Ebs:
              VolumeSize: 8
              VolumeType: gp2
              DeleteOnTermination: true
              Encrypted: true
            DeviceName: /dev/xvdh
        ImageId: ami-098f16afa9edf40be
        InstanceType: t2.micro
        SecurityGroupIds:
          - !GetAtt EC2SecurityGroup.GroupId
DemoAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: demo-auto-scaling-group
      MinSize: "2"
      MaxSize: "4"
      DesiredCapacity: "2"
      HealthCheckGracePeriod: 300
      LaunchTemplate:
        LaunchTemplateId: !Ref DemoLaunchTemplate
        Version: !GetAtt DemoLaunchTemplate.LatestVersionNumber
      VPCZoneIdentifier:
        - subnet-0123
        - subnet-0456

Posted in Uncategorized | Tagged , | Leave a comment

AWS CloudFormation: Target Group does not have an associated Load Balancer

Yesterday I was using an AWS CloudFormation template to ultimately create an ECS Service (Fargate type), but also create resources including an Application Load Balancer, Target Group and IAM Roles.

When the stack was being created, I received the following error:

The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-1:999999995545:targetgroup/MyTargetGroup/999999eb40599999 does not have an associated load balancer. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 5da2a1ed-a216-4666-a6f9-8af18ef37af6)

The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-1:999999995545:targetgroup/MyTargetGroup/999999eb40599999 does not have an associated load balancer. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 54321987-a2a2-4444-abcd-8af18ef12345)

I inspected my template a number of times, and felt it was correct. I found a post on a forum suggesting the error may be due to the fact the Load Balancer may not have been created yet by the time the ECS Service was being created.

The solution: Use the DependsOn attribute for the ECS Service resource.

Here is a portion of my AWS CloudFormation Template, using the DependsOn attribute:

#Create Application Load Balancer
  DemoApplicationLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Type: application
      Name: Demo-ALB
      IpAddressType: ipv4
      Scheme: internet-facing
      # Other properties...

# Create Security Groups, IAM Roles, Load Balancing Listener, ECS Cluster, ECS Task Def, etc.

# Create ECS Service - with DependsOn attribute
  DemoSandboxService:
    Type: AWS::ECS::Service
    DependsOn:
    - DemoLoadBalancerListener
    Properties:
      Cluster:
          Ref: DemoSandboxCluster
      # Other properties...
Posted in Uncategorized | Tagged , , , | Leave a comment

Elasticsearch – Ignore special characters in query with pattern replace filter and custom analyzer

Using Elasticsearch 5, we had a field like drivers license number where values may include special characters and inconsistent upper/lower case behavior as the values were entered by the users with limited validation.  For example, these are hypothetical values:

  • CA-123-456-789
  • WI.12345.6789
  • tx123456789
  • az-123-xyz-456

In our application, the end user need to search by that field. We had a business requirement that user should be able to not have to enter any special characters such as hyphens and periods to get back the record.  So for the first example above, the user should be able to type any of these values and see that record:

  • CA-123-456-789 (an exact match)
  • CA123456789  (no special chars)
  • ca123456789  (lower-case letters and no special chars)
  • Ca.123.456-789 (mixed case letters and mixed special chars)

Our approach was to write a custom analyzer that ignores special characters and then query against that field.

Step 1:  Create pattern replace character filter and custom analyzer

We defined a pattern replace character filter to remove any non-alphanumeric characters as follows on the index:

"char_filter": {
    "specialCharactersFilter": {
        "pattern": "[^A-Za-z0-9]",
        "type": "pattern_replace",
        "replacement": ""
    }
}

Then we used that filter to create a custom analyzer that we named “alphanumericStringAnalyzer” on the index:

"analyzer": {
    "alphanumericStringAnalyzer": {
        "filter": "lowercase",
        "char_filter": [
            "specialCharactersFilter"
        ],
        "type": "custom",
        "tokenizer": "standard"
    }
}

Step 2: Define field mapping using the custom analyzer

The next step was to define a new field mapping that used the new “alphanumericStringAnalyzer” analyzer:

"driversLicenseNumber": {
    "type": "text",
    "fields": {
        "alphanumeric": {
        "type": "text",
            "analyzer": "alphanumericStringAnalyzer"
        },
        "raw": {
            "type": "keyword"
        }
    }
}

Step 3: Run query against new field

In our case, we have this match query as part of a boolean query in the “should” clause:

{
    "match" : {
        "driversLicenseNumber.alphanumeric" : {
            "query" : "Ca.123.456-789",
            "operator" : "OR",
            "boost" : 10.0
        }
    }
}

 

Posted in Elasticsearch, Uncategorized | Tagged | Leave a comment

Elasticsearch 5 Determining if nested field exists

Consider an index with documents such as these, where the team location may or may not be populated:

{
  "status" : "active",
  "name" : "Jimmy Chitwood",
  "team" : {
    "name" : "Hickory Huskers",
    "location" : "Hickory, Indiana"
  } 
}

Note how the document has a nested object “team”.

With Elastic 5.0, I wanted to run a query against this index that would only return documents that have a team.location.  I needed to wrap the “exists” query in a “nested” query.

This is the query I used:

{
  "from": 0,
  "size": 100,
  "query": {
    "bool": {
      "must": [
      {
        "nested": {
          "path": "team",
          "query": {
            "exists": {
              "field": "team.location"
            }
          }
        }
      }
      ]
    }
  }
}

 

Posted in Elasticsearch | Tagged | Leave a comment

Elasticsearch deprecation warning: [deprecation.search.aggregations.bucket.terms] sorting by ascending count is deprecated and will be removed in the next major version

I am currently attempting to upgrade from Elasticsearch 2.4.1 to Elasticsearch 5.  While running ES 2.4.1 I turned on deprecation logging and executed the following search  (which was generated by my application):

GET publicaccess/cases/_search
{
  "from" : 0,
  "size" : 25,
  "query" : {
    "bool" : {
      "must" : {
        "match_all" : { }
      }
    }
  },
  "aggregations" : {
    "caseStatusAgg" : {
      "terms" : {
        "field" : "caseStatus.raw"
      }
    },
  }
}

It caused the following error to show in my deprecation log:

[deprecation.search.aggregations.bucket.terms] sorting by ascending count is deprecated and will be removed in the next major version

The issue is that the terms aggregation does not specify a sort order.  The fix was to add an order to the terms aggregation as follows:

GET publicaccess/cases/_search
{
  "from" : 0,
  "size" : 25,
  "query" : {
    "bool" : {
      "must" : {
        "match_all" : { }
      }
    }
  },
  "aggregations" : {
    "caseStatusAgg" : {
      "terms" : {
        "field" : "caseStatus.raw",
        "order": {
          "_term": "asc"
        }
      }
    },
  }
}
Posted in Elasticsearch | Tagged | 1 Comment

Using Spring to populate Hazelcast Map at Start-up

The application I work on has been using Hazelcast for awhile now, but today we had a new scenario where we wanted to initially populate the Hazelcast Map from a relational database but did not want the our Map to be backed by a data store.  We wanted our Map to continue to only be in memory (for performance reasons).

As a relatively “newbie” to Hazelcast, I initially focused on the MapLoader interface, but as the documentation on that interface explains, it is used when the Map is backed by a data store.

We ended up using Spring to populate our Map by using the init method on our beans.

Here is the relevant code…

First, with our Hazelcast Map the keys will be of type String and the values will be of type ProcessStats.  The ProcessStats class:

public class ProcessStats implements Serializable {
  private static final long serialVersionUID = 1L;
  private String processName;
  private Long total;
 
  public ProcessStats(String processName, Long total) {
    super();
    this.processName = processName;
    this.total= total;
  }

  // Getters and Setters left out for brevity
}

 

Finally, here is the Spring bean that we used to populate the Map.

public class HazelcastCacheLoader {
 HazelcastInstance hz;

 @PostConstruct
 public void init() 
 {
   IMap<String, Object> processStatsMap = hz.getMap("processStats");
   List<Proccess> processes = // Retrieve this list from the database
   for (Process proc : processes)
   {
     String key = proc.getProcessID();
     String processStats = calcStatsForProcess(proc);
     processStatsMap.put(key, processStats);
   }
 }
}

 

Posted in Hazelcast, Spring | Tagged , | Leave a comment

Spring Boot Web Application – Starting in Eclipse

I am going through the Spring Boot Getting Started Tutorial and trying to follow the steps in Eclipse (and not STS).  I made it to the end and was thrown off a little by how to “start” it up.

I wanted to create a Tomcat Server within Eclipse for the project and start.  However the tutorial does not mention that step, instead saying to run the Maven command:

mvn spring-boot:run

I was initially confused as I was wanting to start my Tomcat server within Eclipse, perhaps with some additional start up parameters.  NO!

Instead do the following steps:

  1. In your Project Explorer view, right click and select Run As -> Maven build…
  2. On the Edit Configurations dialog, in the “Goals” field enter:  mvn spring-boot:run
  3. Finally, click Run

That Maven goal will actually start up an embedded Tomcat server (behind the scenes).  It will not use any Tomcat server you have have created in the Eclipse Servers view.

In the end, the tutorial steps are correct, it was just me not following the steps.

 

Posted in Eclipse, Maven, Spring, Spring Boot, Tomcat | Tagged , , , , | 4 Comments

Spring Boot Application and Tomcat error: A child container failed during start

I was attempting to create my first Spring Boot web application this afternoon by following the Spring Boot Getting Started Tutorial on spring.io, yet attempting to do the steps in Eclipse Luna with Java 8 and an embedded Tomcat 7 server.

Upon trying to start the Tomcat 7 server from within Eclipse, I received the following error – Notice hidden in the error stack: Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.getVirtualServerName():

SEVERE: A child container failed during start
java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/MavenWebDemo]]
 at java.util.concurrent.FutureTask.report(FutureTask.java:122)
 at java.util.concurrent.FutureTask.get(FutureTask.java:192)
 at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1123)
 at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:799)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/MavenWebDemo]]
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
 ... 6 more
Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
 at org.apache.tomcat.websocket.server.WsServerContainer.<init>(WsServerContainer.java:147)
 at org.apache.tomcat.websocket.server.WsSci.init(WsSci.java:131)
 at org.apache.tomcat.websocket.server.WsSci.onStartup(WsSci.java:47)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5456)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 ... 6 more

After some searching, I found the solution buried in the Spring Boot documentation for how to embed Servlet containers.  It was as simple as adding a “tomcat.version” property to your Maven project’s pom.xml file:

<properties>
    <tomcat.version>7.0.52</tomcat.version>
</properties>
<dependencies>
    ...

 

Spring Boot defaults to Tomcat 8, so I needed the tomcat.version property to tell Spring Boot that I intend to target Tomcat 7.

I restarted my server and it started right up!

 

Posted in Java 8, Maven, Spring, Spring Boot, Tomcat | Tagged , , , , | 7 Comments

Drupal Superfish Menu is rendering without styles

I am upgrading a Drupal website I administer and decided to install the Superfish module for the first time.  As part of the installation, I also uploaded the Superfish-for-Drupal-master.zip to my /sites/all/libraries directory and then extracted all the files.  I then went through the rest of the install steps, including enabling a Superfish block.

However, when I went to view my menu on my website, the menu options and links were all there but there was no menu-like styling or behavior.  I inspected the HTML using Chrome Developer Tools and each of the menu link elements had Superfish CSS classes added to them.  The problem was the CSS files themselves were not being found.

The solution was simply to rename the Superfish directory in my /sites/all/libraries directory from “Superfish-for-Drupal-master” to “Superfish”.  I still feel like this is a missing step in the installation steps (or at a minimum isn’t called out as it should).

I hope this helps someone else as I spent way too much time on this!

Posted in Drupal | Tagged | Leave a comment

Using Matches for null parameter with Mockito

I was using Mockito to write a unit test today and had a brief struggle trying to write a Matcher for a parameter I expected to be null.

Here is the method in my interface that was causing problems:

public interface ItemService
{
    List<Item> findItems(String query, Map<String, String> qualifiers);
}

In my unit test I started with writing this for argument verification since I expected the second parameter for the Map to be null:

 verify(itemService).findItems(anyString(), null);

This resulted in the following error when running the unit test:

org.mockito.exceptions.misusing.InvalidUseOfMatchersException: 
Invalid use of argument matchers!
2 matchers expected, 1 recorded:
-> at com.stevewall123.ItemServiceTest.testFindITems(ItemServiceTest.java:35)

The error is pretty straight-forward.  Once you use a Matcher on one argument then they need to be used on all arguments.  To fix the error, I needed to use the Matchers class isNull() method.  This method will verify the argument is null.  So in my unit test, I changed the argument verification line to:

 verify(itemService).findItems(anyString(), isNull(Map.class);

And now the test passes!

Posted in Java, Mockito | Tagged , | Leave a comment