Amazon EC2 error while launching instances The maximum number of VPCs has been reached

I am trying to create a Python program using boto3 that creates an Amazon AutoScaling group. Define scale up and scale down policies and corresponding CloudWatch alarms. Supply the startup script created in previous step while launching a new instance from the program. Using a security group port 80 open, here is the program ,

# Check if the user has the Access & Secret key configured import boto3 from boto3 import Session session = Session() credentials = session.get_credentials() current_credentials = credentials.get_frozen_credentials() # Break & Exit if any of the key is not present if current_credentials.access_key is None: print("Access Key missing, use `aws configure` to setup") exit() if current_credentials.secret_key is None: print("Secret Key missing, use `aws configure` to setup") exit() # VPC design for multi az deployments globalVars = {} globalVars['REGION_NAME'] = "ap-south-1" globalVars['AZ1'] = "ap-south-1a" globalVars['AZ2'] = "ap-south-1b" globalVars['CIDRange'] = "10.240.0.0/23" globalVars['az1_pvtsubnet_CIDRange'] = "10.240.0.0/25" globalVars['az1_pubsubnet_CIDRange'] = "10.240.0.128/26" globalVars['az1_sparesubnet_CIDRange'] = "10.240.0.192/26" globalVars['az2_pvtsubnet_CIDRange'] = "10.240.1.0/25" globalVars['az2_pubsubnet_CIDRange'] = "10.240.1.128/26" globalVars['az2_sparesubnet_CIDRange'] = "10.240.1.192/26" globalVars['Project'] = { 'Key': 'Name', 'Value': 'test1'} globalVars['tags'] = [{'Key': 'Owner', 'Value': 'test1'}, {'Key': 'Environment', 'Value': 'Test'}, {'Key': 'Department', 'Value': 'TestD'}] # EC2 Parameters globalVars['EC2-Amazon-AMI-ID'] = "ami-d783a9b8" globalVars['EC2-InstanceType'] = "t2.micro" globalVars['EC2-KeyName'] = "datastructutre key" # AutoScaling Parameters globalVars['ASG-LaunchConfigName'] = "ASG-Demo-LaunchConfig" globalVars['ASG-AutoScalingGroupName'] = "ASG-Demo-AutoScalingGrp" # Creating a VPC, Subnet, and Gateway ec2 = boto3.resource('ec2', region_name=globalVars['REGION_NAME']) ec2Client = boto3.client('ec2', region_name=globalVars['REGION_NAME']) vpc = ec2.create_vpc(CidrBlock=globalVars['CIDRange']) asgClient = boto3.client('autoscaling', region_name=globalVars['REGION_NAME']) rds = boto3.client('rds', region_name=globalVars['REGION_NAME']) # AZ1 Subnets az1_pvtsubnet = vpc.create_subnet(CidrBlock=globalVars['az1_pvtsubnet_CIDRange'], AvailabilityZone=globalVars['AZ1']) az1_pubsubnet = vpc.create_subnet(CidrBlock=globalVars['az1_pubsubnet_CIDRange'], AvailabilityZone=globalVars['AZ1']) az1_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az1_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ1']) # AZ2 Subnet az2_pvtsubnet = vpc.create_subnet(CidrBlock=globalVars['az2_pvtsubnet_CIDRange'], AvailabilityZone=globalVars['AZ2']) az2_pubsubnet = vpc.create_subnet(CidrBlock=globalVars['az2_pubsubnet_CIDRange'], AvailabilityZone=globalVars['AZ2']) az2_sparesubnet = vpc.create_subnet(CidrBlock=globalVars['az2_sparesubnet_CIDRange'], AvailabilityZone=globalVars['AZ2']) # Enable DNS Hostnames in the VPC vpc.modify_attribute(EnableDnsSupport={'Value': True}) vpc.modify_attribute(EnableDnsHostnames={'Value': True}) # Create the Internet Gatway & Attach to the VPC intGateway = ec2.create_internet_gateway() intGateway.attach_to_vpc(VpcId=vpc.id) # Create another route table for Public & Private traffic routeTable = ec2.create_route_table(VpcId=vpc.id) rtbAssn=[] rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pubsubnet.id)) rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az1_pvtsubnet.id)) rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pubsubnet.id)) rtbAssn.append(routeTable.associate_with_subnet(SubnetId=az2_pvtsubnet.id)) # Create a route for internet traffic to flow out intRoute = ec2Client.create_route(RouteTableId=routeTable.id, DestinationCidrBlock='0.0.0.0/0', GatewayId=intGateway.id) # Tag the resources vpc.create_tags (Tags=globalVars['tags']) az1_pvtsubnet.create_tags (Tags=globalVars['tags']) az1_pubsubnet.create_tags (Tags=globalVars['tags']) az1_sparesubnet.create_tags(Tags=globalVars['tags']) az2_pvtsubnet.create_tags (Tags=globalVars['tags']) az2_pubsubnet.create_tags (Tags=globalVars['tags']) az2_sparesubnet.create_tags(Tags=globalVars['tags']) intGateway.create_tags (Tags=globalVars['tags']) routeTable.create_tags (Tags=globalVars['tags']) vpc.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-vpc'}]) az1_pvtsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-private-subnet'}]) az1_pubsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-public-subnet'}]) az1_sparesubnet.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az1-spare-subnet'}]) az2_pvtsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-private-subnet'}]) az2_pubsubnet.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-public-subnet'}]) az2_sparesubnet.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-az2-spare-subnet'}]) intGateway.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-igw'}]) routeTable.create_tags (Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-rtb'}]) # Let create the Public & Private Security Groups elbSecGrp = ec2.create_security_group(DryRun=False, GroupName='elbSecGrp', Description='ElasticLoadBalancer_Security_Group', VpcId=vpc.id ) pubSecGrp = ec2.create_security_group(DryRun=False, GroupName='pubSecGrp', Description='Public_Security_Group', VpcId=vpc.id ) pvtSecGrp = ec2.create_security_group(DryRun=False, GroupName='pvtSecGrp', Description='Private_Security_Group', VpcId=vpc.id ) elbSecGrp.create_tags(Tags=globalVars['tags']) pubSecGrp.create_tags(Tags=globalVars['tags']) pvtSecGrp.create_tags(Tags=globalVars['tags']) elbSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-elb-security-group'}]) pubSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-public-security-group'}]) pvtSecGrp.create_tags(Tags=[{'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-private-security-group'}]) # Add a rule that allows inbound SSH, HTTP, HTTPS traffic ( from any source ) ec2Client.authorize_security_group_ingress(GroupId=elbSecGrp.id, IpProtocol='tcp', FromPort=80, ToPort=80, CidrIp='0.0.0.0/0' ) # Allow Public Security Group to receive traffic from ELB Security group ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id, IpPermissions=[{'IpProtocol': 'tcp', 'FromPort': 80, 'ToPort': 80, 'UserIdGroupPairs': [{'GroupId': elbSecGrp.id}] }] ) # Allow Private Security Group to receive traffic from Application Security group ec2Client.authorize_security_group_ingress(GroupId=pvtSecGrp.id, IpPermissions=[{'IpProtocol': 'tcp', 'FromPort': 3306, 'ToPort': 3306, 'UserIdGroupPairs': [{'GroupId': pubSecGrp.id}] }] ) ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id, IpProtocol='tcp', FromPort=80, ToPort=80, CidrIp='0.0.0.0/0' ) ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id, IpProtocol='tcp', FromPort=443, ToPort=443, CidrIp='0.0.0.0/0' ) ec2Client.authorize_security_group_ingress(GroupId=pubSecGrp.id, IpProtocol='tcp', FromPort=22, ToPort=22, CidrIp='0.0.0.0/0' ) # The user defined code to install WebServer & Configure them userDataCode = """ #!/bin/bash set -e -x # Setting up the HTTP server yum install -y httpd service httpd start chkconfig httpd on groupadd www usermod -a -G www ec2-user cd /var/www/ # Set the permissions chown -R root:www /var/www chmod 2775 /var/www find /var/www -type d -exec chmod 2775 {} + find /var/www -type f -exec chmod 0664 {} + # SE Linux permissive # setsebool -P httpd_can_network_connect=1 service httpd restart # Remove below file after testing echo "<?php phpinfo(); ?>" > /var/www/html/phptestinfo.php """ # Create the Public Instance ##### **DeviceIndex**:The network interface's position in the attachment order. For example, the first attached network interface has a DeviceIndex of 0 instanceLst = ec2.create_instances(ImageId=globalVars['EC2-Amazon-AMI-ID'], MinCount=1, MaxCount=1, KeyName="datastructutre key", UserData=userDataCode, InstanceType=globalVars['EC2-InstanceType'], NetworkInterfaces=[ { 'SubnetId': az1_pubsubnet.id, 'Groups': [pubSecGrp.id], 'DeviceIndex': 0, 'DeleteOnTermination': True, 'AssociatePublicIpAddress': True, } ] ) # Create the Launch Configuration # InstanceId = 'string' asgLaunchConfig = asgClient.create_launch_configuration( LaunchConfigurationName=globalVars['ASG-LaunchConfigName'], ImageId=globalVars['EC2-Amazon-AMI-ID'], KeyName=globalVars['EC2-KeyName'], SecurityGroups=[pubSecGrp.id], UserData=userDataCode, InstanceType=globalVars['EC2-InstanceType'], InstanceMonitoring={'Enabled': False }, EbsOptimized=False, AssociatePublicIpAddress=False ) # create Auto-Scaling Group ASGSubnets = az1_pubsubnet.id + "," +az2_pubsubnet.id asGroup=asgClient.create_auto_scaling_group( AutoScalingGroupName=globalVars['ASG-AutoScalingGroupName'], LaunchConfigurationName=globalVars['ASG-LaunchConfigName'], MinSize=1, MaxSize=3, DesiredCapacity=2, DefaultCooldown=120, HealthCheckType='EC2', HealthCheckGracePeriod=60, Tags=globalVars['tags'], VPCZoneIdentifier=ASGSubnets ) asgClient.create_or_update_tags( Tags=[ { 'ResourceId': globalVars['ASG-AutoScalingGroupName'], 'ResourceType': 'auto-scaling-group', 'Key': 'Name', 'Value': globalVars['Project']['Value'] + '-ASG-Group', 'PropagateAtLaunch': True }, ] ) ###### Print to Screen ######## print("VPC ID : {0}".format(vpc.id)) print("AZ1 Public Subnet ID : {0}".format(az1_pubsubnet.id)) print("AZ1 Private Subnet ID : {0}".format(az1_pvtsubnet.id)) print("AZ1 Spare Subnet ID : {0}".format(az1_sparesubnet.id)) print("Internet Gateway ID : {0}".format(intGateway.id)) print("Route Table ID : {0}".format(routeTable.id)) print("Public Security Group ID : {0}".format(pubSecGrp.id)) print("Private Security Group ID : {0}".format(pvtSecGrp.id)) print("EC2 Key Pair : {0}".format(globalVars['EC2-KeyName'])) print("EC2 PublicIP : {0}".format(globalVars['EC2-KeyName'])) print("RDS Endpoint : {0}".format(globalVars['Endpoint'])) ###### Print to Screen ######## """ Function to clean up all the resources """ def cleanAll(resourcesDict=None): # Delete the instances ids = [] for i in instanceLst: ids.append(i.id) ec2.instances.filter(InstanceIds=ids).terminate() # Wait for the instance to be terminated waiter = ec2Client.get_waiter('instance_terminated') waiter.wait(InstanceIds=[ids]) ec2Client.delete_key_pair(KeyName=globalVars['EC2-KeyName']) # Delete Routes & Routing Table for assn in rtbAssn: ec2Client.disassociate_route_table(AssociationId=assn.id) routeTable.delete() # Delete Subnets az1_pvtsubnet.delete() az1_pubsubnet.delete() az1_sparesubnet.delete() # Detach & Delete internet Gateway ec2Client.detach_internet_gateway(InternetGatewayId=intGateway.id, VpcId=vpc.id) intGateway.delete() # Delete Security Groups pubSecGrp.delete() pvtSecGrp.delete() vpc.delete() 

how ever I have not reached that stage to write scale up and scale down policies for cloud watch alarms in the above code, it is just that while executing it before proceeding further I got following errors,

autoscaling.py", line 51, in <module> vpc = ec2.create_vpc(CidrBlock=globalVars['CIDRange']) File "E:\installation2\python3\lib\site-packages\boto3\resources\factory.py", line 520, in do_action response = action(self, *args, **kwargs) File "E:\installation2\python3\lib\site-packages\boto3\resources\action.py", line 83, in __call__ response = getattr(parent.meta.client, operation_name)(**params) File "E:\installation2\python3\lib\site-packages\botocore\client.py", line 314, in _api_call return self._make_api_call(operation_name, kwargs) File "E:\installation2\python3\lib\site-packages\botocore\client.py", line 612, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (VpcLimitExceeded) when calling the CreateVpc operation: The maximum number of VPCs has been reached. 

What should I do to get rid of this error :The maximum number of VPCs has been reached.
, I use free tier services of amazon.

The output of aws ec2 describe-security-groups is following

 { "SecurityGroups": [ { "Description": "default VPC security group", "GroupName": "default", "IpPermissions": [ { "IpProtocol": "-1", "IpRanges": [], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [ { "GroupId": "sg-03c0e0d31aca5827b", "UserId": "101010101010" } ] } ], "OwnerId": "101010101010", "GroupId": "sg-03c0e0d31aca5827b", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-06eedbb5dc8c8e20b" }, { "Description": "default VPC security group", "GroupName": "default", "IpPermissions": [ { "IpProtocol": "-1", "IpRanges": [], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [ { "GroupId": "sg-080f42f6c90e60956", "UserId": "101010101010" } ] } ], "OwnerId": "101010101010", "GroupId": "sg-080f42f6c90e60956", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-0a0a0699b064d3382" }, { "Description": "default VPC security group", "GroupName": "default", "IpPermissions": [ { "IpProtocol": "-1", "IpRanges": [], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [ { "GroupId": "sg-096d48a3a161a98cc", "UserId": "101010101010" } ] } ], "OwnerId": "101010101010", "GroupId": "sg-096d48a3a161a98cc", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-090b6525d5e4166bd" }, { "Description": "default VPC security group", "GroupName": "default", "IpPermissions": [ { "IpProtocol": "-1", "IpRanges": [], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [ { "GroupId": "sg-0aea8c83e780f8bca", "UserId": "101010101010" } ] } ], "OwnerId": "101010101010", "GroupId": "sg-0aea8c83e780f8bca", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-0cf43d41666c21f82" }, { "Description": "launch-wizard-1 created 2018-08-01T09:57:51.176+05:30", "GroupName": "launch-wizard-1", "IpPermissions": [ { "FromPort": 80, "IpProtocol": "tcp", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [ { "CidrIpv6": "::/0" } ], "PrefixListIds": [], "ToPort": 80, "UserIdGroupPairs": [] }, { "FromPort": 22, "IpProtocol": "tcp", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "ToPort": 22, "UserIdGroupPairs": [] } ], "OwnerId": "101010101010", "GroupId": "sg-0e81c2a33e1039f58", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-08356c60" }, { "Description": "default VPC security group", "GroupName": "default", "IpPermissions": [ { "FromPort": 80, "IpProtocol": "tcp", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [ { "CidrIpv6": "::/0" } ], "PrefixListIds": [], "ToPort": 80, "UserIdGroupPairs": [] }, { "FromPort": 22, "IpProtocol": "tcp", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "ToPort": 22, "UserIdGroupPairs": [] } ], "OwnerId": "101010101010", "GroupId": "sg-40e5492a", "IpPermissionsEgress": [ { "IpProtocol": "-1", "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [], "UserIdGroupPairs": [] } ], "VpcId": "vpc-08356c60" } ] } 

I have 6 vpc all mentioning defalt logged in to web console don't know command line output to find them here are they

vpc-06eedbb5dc8c8e20b vpc-0a0a0699b064d3382 vpc-090b6525d5e4166bd vpc-0cf43d41666c21f82 launch-wizard-1 created 2018-08-01T09:57:51.176+05:30 vpc-0a0a0699b064d3382 

the description of all of them reads Default VPC security group. Basically what I am trying to achieve is to create an auto-scaling configuration for the web tier of a cloud application. Following the steps below.  Create a static website with one or two HTML pages and copy the pages and dependent files in an Amazon S3 bucket.  Create a startup script to install Apache server and copy the website files from S3 to the instance.  Create a Python program using boto that creates an Amazon AutoScaling group. Define scale up and scale down policies and corresponding CloudWatch alarms. Supply the startup script you created in previous step while launching a new instance from the program. Use a security group port 80 open.  Open the public DNS of the newly launched instance in a browser and verify if the static website works. What I have not been able to do above is set cloudwatch alarms and metrics and see how they run in my code.

1 Answer

You don't want to create a new VPC each time. So get rid of your ec2.create_vpc call. Instead just define your vpc variable from a describe vpcs call (don't know what that is off the top of my head in boto3) - probably something like ec2.describe_vpcs -> then select whatever one you want to use. Or just hard code the vpc id in your script.

@Kush is correct - there is a soft limit of 5 VPCs per region but you don't really need to create new VPCs for everything you do. If you need to segment different aspects of you AWS infrastructure for security reasons you can rely on subnet ACLs and security groups to accomplish this.