tag:blogger.com,1999:blog-43300343783774631732024-03-05T13:03:38.239+07:00Random StuffVarious ramblings of sysadmin, programmer, dancer, coffee snob, food lover and Winnipegger.testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comBlogger78125tag:blogger.com,1999:blog-4330034378377463173.post-69239294979344998762017-07-12T16:00:00.000+07:002017-07-16T18:07:06.491+07:00Terraform G Suite DNS records<div dir="ltr" style="text-align: left;" trbidi="on">In the <a href="http://blog.shvetsov.com/2017/07/why-transfer-domains-from-namecheap-to-route-53.html">previous post</a> I cited <em>automation</em> potential as one of the reasons to change registrars. While it’s a pretty strong reason for cloud admins, it might be a bit obscure for others…<br />
<br />
Back in the day Google offered <a href="https://support.google.com/a/answer/2855120">free edition</a> of G Suite (Google Apps for Domain). As I acquired domains and corresponding free G Suite licenses, each required creation of various DNS records for integration. These days G Suite is not free, but plenty of organisations use it and must go through the same process. <a href="https://en.wikipedia.org/wiki/Infrastructure_as_Code">IaC</a> is a passion of mine, so I decided to hone my <a href="https://www.terraform.io/">Terraform</a> skills and write a <a href="https://github.com/sshvetsov/tf-gsuite-cloudflare">module</a> for creation and management of G Suite DNS records. For now this module relies on <a href="http://cloudflare.com/">Cloudflare</a> and their free DNS hosting, in the future I might add support for other DNS hosting services like Route 53.<br />
<a name='more'></a><br />
<br />
When building a tool that can do a variety of things, one of the challenging parts is deciding on its default behaviour. Everyone recommends “sane defaults”, but how to decide what's sane? To make life of your end users easier, you have to understand them and their most common uses for your tool. Some users might want only Gmail with their domain; some just want to point their domain to their blog; others want to run their entire company on G Suite with Gmail, Calendar, Contacts, Drive, Calendar, Hangouts, Sites, etc.<br />
<br />
I decided not to make assumptions about the most common case and left all features off by default. The only required variable is <code>domain_name</code> of your Cloudflare site (which must be already added to your account). Run <code>terraform apply</code> on the following config and it will do absolutely nothing:<br />
<pre>module "my_domain" {
source = "github.com/sshvetsov/tf-gsuite-cloudflare"
domain_name = "yourdomain.com"
}</pre><br />
Want to create a <a href="https://support.google.com/a/answer/174125">set of MX records for Gmail</a>? Add one more variable:<br />
<pre>use_default_mx_records = true</pre><br />
Want to add <a href="https://support.google.com/a/answer/178723">recommended TXT record for SPF</a>? Add another variable:<br />
<pre>use_default_spf_record = true</pre><br />
Want to prevent spoofing by signing your emails with <a href="https://support.google.com/a/answer/173535">Google generated DKIM key</a>? Easy:<br />
<pre>dkim_value = "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."</pre><br />
If secure Gmail is all you’re using G Suites for, similar config should be more than enough:<br />
<pre>module "my_domain" {
source = "github.com/sshvetsov/tf-gsuite-cloudflare"
domain_name = "yourdomain.com"
use_default_mx_records = true
use_default_spf_record = true
dkim_value = "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."
}</pre><br />
Want your root/apex/bare domain <code>yourdomain.com</code> <a href="https://support.google.com/a/answer/2518373">redirecting</a> to <code>blog.yourdomain.com</code>, your <a href="https://support.google.com/blogger/answer/55374">custom Blogger domain</a>? Voila:<br />
<pre>module "my_domain" {
source = "github.com/sshvetsov/tf-gsuite-cloudflare"
domain_name = "yourdomain.com"
redirect_apex = true
custom_cname_names = ["blog", "jlryza01k1ls"]
custom_cname_values = ["ghs.google.com", "gv-bgzglgxbgymvbz.dv.googlehosted.com"]
}</pre><br />
Want your Drive, Calendar and Mail to have matching <a href="https://support.google.com/a/answer/53340">subdomains</a>? Just add this variable:<br />
<pre>create_app_cname_records = true</pre><br />
Don’t like my name choices? Want to subdomain to more G Suite services? Sure, just pass the list:<br />
<pre>app_cname_names = [
"webmail",
"cal",
"docs",
"site-a",
"site-b",
]</pre><br />
Other options are available and almost everything could be customised. Take a look at the examples provided in the <a href="https://github.com/sshvetsov/tf-gsuite-cloudflare">repo</a> to get a sense of other possibilities.<br />
<br />
<em>Note: I started writing my module as an exercise and only bothered looking around for prior work after it was finished. Surprisingly, my search yielded only one <a href="https://github.com/gnarea/terraform-gmail">similar project</a>, so hopefully my module will be of benefit to larger community. Enjoy!</em></div>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-53296517522661518142017-07-02T13:59:00.002+07:002017-07-03T20:30:17.341+07:00Why transfer domains from NameCheap to Route 53<div dir="ltr" style="text-align: left;" trbidi="on">
TL;DR: it’s cheaper if you want WhoisGuard and use Route 53 as registrar <strong>only</strong>. Longer answer with caveats and additional justifications below.<br />
<a name='more'></a><br />
<h4>
Price comparison</h4>
From the purely financial point of view, if you’re planning to hold on to a <code>.com</code> domain for more than a year and wish to protect your contact information with <a href="https://www.namecheap.com/support/knowledgebase/article.aspx/278/37/what-is-whoisguard">WhoisGuard</a>, NameCheap will cost you:<br />
<pre>$13.75 => $10.69 (renewal) + $0.18 (ICANN fee) + $2.88 (WhoisGuard)</pre>
Buying a <code>.com</code> from Amazon Route 53 Domains currently costs:<br />
<pre>$12.00 => already including ICANN fee and Whois privacy</pre>
<h4>
Caveats</h4>
<blockquote>
If you want privacy and cost savings offered by Amazon Route 53 Domains, you must use free 3rd party DNS hosting (<a href="https://www.cloudflare.com/">Cloudflare</a>, <a href="https://www.namecheap.com/domains/freedns.aspx">NameCheap</a>, etc.) and watch out for your local sales taxes.</blockquote>
<ol style="text-align: left;">
<li>If your billing address is in jurisdiction requiring Amazon to charge you sales tax (VAT, GST, HST, PST, etc.), you will have to pay that on top. Check <a href="https://aws.amazon.com/tax-help/">what taxes may apply</a> to you first.</li>
<li>If you’re not using your hosting provider’s name servers, you will need to setup a <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/AboutHZWorkingWith.html">public hosted zone</a> in Route 53, which will cost you around <a href="https://aws.amazon.com/route53/pricing/">$0.50 per zone per month</a>.</li>
<li>Any DNS queries resolved by the hosted zone will cost you under <a href="https://aws.amazon.com/route53/pricing/">$0.04 per 100,000 queries per month</a>.</li>
</ol>
<h4>
Justifications</h4>
<blockquote>
Switching to Route 53 offers better security, better record management API, better portability, advanced DNS features, great price and similar no-nonsense renewal process — ideal for seasoned sysadmins.</blockquote>
I originally moved my domains from GoDaddy to NameCheap because of great prices and no-nonsense renewal process. So why go through effort of moving your domain from NameCheap to Route 53, and potentially having to setup separate DNS hosting account only to save <code>$1.75</code> per year?<br />
<ol>
<li><strong>Automation</strong>. NameCheap’s API is not very good for managing large number of records. Cloudflare and Route 53 APIs are, and they are also very well supported by <a href="https://en.wikipedia.org/wiki/Infrastructure_as_Code">Infrastructure as Code</a> tools like <a href="https://www.ansible.com/">Ansible</a> and <a href="https://www.terraform.io/">Terraform</a>.</li>
<li><strong>Advanced DNS features</strong>. If you’re managing a more complex cloud infrastructure, you might want to pay a bit more for hosted zone and gain additional Route 53 features, like DNS routing policies, health checks and failover.</li>
<li><strong>Portability</strong>. Surprisingly, transferring domains from NameCheap took more effort than from GoDaddy! I had to open a support ticket to export my zone files, and NameCheap confirmation emails and docs implied that it takes 5 days for domain transfer to complete, while in reality it can be done on the same day.</li>
<li><strong>Price</strong>. $1.75 could add up to a sizeable amount when you have multiple domain names.</li>
<li><strong>Better security</strong>. NameCheap supports only SMS-based <a href="https://en.wikipedia.org/wiki/Multi-factor_authentication">MFA</a>, which is easy to intercept and therefore insecure. I don’t know why NameCheap still hasn’t added support for software tokens like Google Authenticator or <a href="https://authy.com/">Authy</a>.</li>
</ol>
</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-10668289415838489442017-04-03T00:40:00.000+07:002017-04-03T00:40:46.361+07:00Rebuild AMI using SSM Automation<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnDmUfZYO1DMWQJpjl3p0vt4JtlQeqUYECikTTFcVv6fIK4YkzJwH4jm_ZWzZttzFLcQTiGhRqEXFGH6pXPJnPMe0iA1duxkFPdF_a3JuqU8inGF2GEcPA4feA3ZTN1aTZChbX4r6IMQc/s1600/awscli-150x150.png" /></div>
In the <a href="http://blog.shvetsov.com/2017/03/ensure-ec2-auto-recovery-by-rebuilding-ami-with-packer.html">previous post</a> I used <a href="https://www.packer.io/">Packer</a> to remove block device mappings from Ubuntu AMI to ensure EC2 auto-recovery is working. While Packer is a fantastic tool with many features, it is synchronous in its nature and requires active SSH connection to the temporary instance in order to bake new AMI.<br />
<br />
Can we get similar results with AWS tools alone? Indeed, we can!<br />
<br />
In December of 2016, at the <a href="https://aws.amazon.com/new/reinvent/#management-tools">re:Invent</a>, AWS announced a <a href="https://aws.amazon.com/blogs/aws/ec2-systems-manager-configure-manage-ec2-and-on-premises-systems/">slew of interesting new features</a> for their <a href="https://aws.amazon.com/ec2/systems-manager/">Amazon EC2 Systems Manager</a> (a.k.a. SSM). The initial documentation and tutorials were sparse, so their capabilities weren't immediately clear. One of these new features, Automation, offers functionality similar to Packer, so I decided to learn more about it by applying to my simple <a href="http://blog.shvetsov.com/2017/03/ensure-ec2-auto-recovery-by-rebuilding-ami-with-packer.html">use case</a>.<br />
<a name='more'></a><br />
Start by <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-ami-permissions.html">configuring IAM Roles for Automation</a> as described in the docs. I know this might seem like an extra step compared to Packer example, there I just used my personal credentials. However, if you serious about security and live by <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">POLP</a>, you should create a dedicated IAM Role or User for Packer as well.<br />
<br />
Next, we will need an automation document - a JSON document defining meta data, parameters, outputs and steps to execute (rather similar to CloudFormation). AWS docs already provide a great <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-ami-createdoc.html">example walk through</a> for a very similar use case, which can be used as a starting point. More detailed information can be found under <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-ami-actions.html">Systems Manager Automation Actions</a> and <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-ami-variables.html">Automation System Variables</a>.<br />
<br />
Here's my automation document for this use case: <a href="https://github.com/sshvetsov/ssm-ubuntu-rebake/blob/master/TVLK-ami-sans-bdm.json">https://github.com/sshvetsov/ssm-ubuntu-rebake/blob/master/TVLK-ami-sans-bdm.json</a>. While I tried to simulate <a href="https://github.com/sshvetsov/packer-ubuntu-rebake/blob/master/ubuntu-14.04-x86_64.json">Packer template</a> from previous post as much as possible, I could not re-create '<code>source_ami_filter</code>' property, which automatically found latest Ubuntu AMI for me. I suspect AWS will advise I write a Lambda instead (sigh)...<br />
<br />
Now we need to load our Automation doc into SSM. These steps can be done via AWS Console, but I decided to do it the hard(?) way, with CLI.<br />
<br />
Create automation document in EC2 Systems Manager (SSM):<br />
<br />
<pre>aws ssm \
create-document \
--name "TVLK-ami-sans-bdm" \
--content "file://TVLK-ami-sans-bdm.json" \
--document-type "Automation"</pre>
<br />
When successful, you should see JSON output with document description, type, version and other meta data.<br />
<br />
If you need to make a change, update your document by creating a new version with:<br />
<br />
<pre>aws ssm \
update-document \
--name "TVLK-ami-sans-bdm" \
--content "file://TVLK-ami-sans-bdm.json" \
--document-version "\$LATEST"</pre>
<br />
<i>Note: while I applaud AWS for incorporating versioning of documents and Lambdas directly into AWS, I would rather have better integration with popular VCS like Git.</i><br />
<br />
List all documents you have created so far with:<br />
<br />
<pre>aws ssm \
list-documents \
--document-filter-list "key=Owner,value=self"</pre>
<br />
To view specific document:<br />
<br />
<pre>aws ssm \
get-document \
--name "TVLK-ami-sans-bdm"</pre>
<br />
In case you want to get your JSON back out of the document without meta data and escape characters, you can use this little <a href="https://stedolan.github.io/jq/">jq</a> trick:<br />
<br />
<pre>aws ssm \
get-document \
--name "TVLK-ami-sans-bdm" \
--query 'Content' \
| jq '.|fromjson'</pre>
<br />
Now we can finally execute our Automation with the following command (just don't forget to substitute real values for <code>AutomationAssumeRole</code>, <code>SubnetId</code> and <code>SourceAmiId</code>):<br />
<br />
<pre>aws ssm \
start-automation-execution \
--document-name "TVLK-ami-sans-bdm" \
--parameters \
"AutomationAssumeRole=arn:aws:iam::123456789012:role/YourAutomationRole,SubnetId=subnet-12345678,SourceAmiId=ami-XXXXXX"</pre>
<br />
If all is well, you should get back <code>AutomationExecutionId</code> as part of the output. Note this value, you will need it to poll status of your automation run from CLI with (again, substitute the real value for <code>--automation-execution-id</code>):<br />
<br />
<pre>aws ssm \
get-automation-execution \
--automation-execution-id "12345678-1234-1234-1234-1234567890ab"</pre>
<br />
Alternatively, just use EC2 Console to monitor the execution.<br />
<br />
In conclusion, here's my thoughts on SSM Automation so far:<br />
<br />
<div style="text-align: left;">
<b>Pros</b>:</div>
<div style="text-align: left;">
<ul style="text-align: left;">
<li>No need to connect to temporary instance.</li>
<li>It's asynchronous, thus better choice in situations where you don't want to wait for the job to finish.</li>
<li>Could be triggered from Lambda or used as a target for <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html">CloudWatch Events</a>.</li>
<li>Can use values from <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html">SSM Parameter Store</a>.</li>
<li>Might be preferred by purists, who want to rely <b>exclusively</b> on AWS tools and services. </li>
</ul>
</div>
<div style="text-align: left;">
<b>Cons</b>:</div>
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li>It's SLOW. It appears that Automation polls for status change of each step at about 2 minute intervals, which results in much slower execution compared to Packer (in this case: ~11min vs ~2min). Packer seems to poll for events more often, thus moves along faster.</li>
<li>It's still new and BUGGY. At the time of this writing I found and reported a bug, which caused interpolation of global variable <code>global:REGION</code> inside 'BuildRegion' tag to fail. If you run this example before the bug is fixed, you should see error "<i>Parameter: global:REGION is not defined in the Automation Document</i>" in the output of the final step.</li>
</ul>
</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-19019291465700327912017-03-20T01:34:00.005+07:002017-03-25T13:25:25.262+07:00Ensure EC2 auto-recovery by rebuilding AMI with Packer<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicfpmiCi7mLqdqciaimO3V5xHA4vRIMyI2Ni47Noa9msWu1AhUG4mRDcuiP0Cba6aG52BKoy6OcTd4GLFFfCiUrMbW4ZHJKuQ7qzNAbnvqwxYB867q5C5l9Lu8yOfwIFHXTWg3l6QCGfY/s1600/ec2-packer-150.png" /></div>
<b>Problem</b>: Amazon EC2 auto recovery fails if machine image has ephemeral block device mappings defined (even when no instance store volumes are attached).<br />
<br />
<b>Solutions</b>:<br />
<ol style="text-align: left;">
<li>Specify custom block device mapping during instance launch.</li>
<li>Rebuild Ubuntu AMI (with automation).</li>
</ol>
This post shows how to achieve the later using <a href="https://www.packer.io/">Packer</a> by <a href="https://www.hashicorp.com/">HashiCorp</a>.<br />
<a name='more'></a><br />
<b>TL;DR</b><br />
<b><br />
</b> I don't like to be woken up at 3 AM to restart a server, especially if hardware of the cloud provider running your instances decides to misbehave. <a href="https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/">Back in 2015</a> AWS made it possible to automatically recover EC2 instances from issues like:<br />
<ul style="text-align: left;">
<li>Loss of network connectivity</li>
<li>Loss of system power</li>
<li>Software issues on the physical host</li>
<li>Hardware issues on the physical host that impact network reachability</li>
</ul>
All you have to do is <i>create a CloudWatch alarm for the metric <b>StatusCheckFailed_System</b> and choose the <b>Recover this instance action</b></i>.<br />
<br />
With some not unreasonable <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html">limitations</a> of course:<br />
<ul style="text-align: left;">
<li>Use a C3, C4, M3, M4, R3, R4, T2, or X1 instance type</li>
<li>Run in a VPC (not EC2-Classic)</li>
<li>Use shared tenancy (not dedicated hardware)</li>
<li>Use EBS volumes (not ephemeral instance store volumes)</li>
</ul>
One limitations is not mentioned in the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html">docs</a> or <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstanceRecovery.html">troubleshooting guide</a> at the time of this writing, however - you cannot auto-recover instance created from AMI defining ephemeral (a.k.a. <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html">instance store</a>) block device mappings. It doesn't matter whether you actually attach ephemeral volumes to your instance, or that instance family doesn't support instance store volumes (M4). <b>If your AMI defines ephemeral devices, auto-recovery fails.</b><br />
<br />
So if you're using official Ubuntu AMIs, you have a problem...<br />
<br />
Let's examine the latest AMI of Ubuntu Trusty server from Canonical (HVM, EBS backed, x86_64). Run the following command (feel free to replace <code>--region</code> value with one of your preference):<br />
<br />
<pre>aws ec2 \
describe-images \
--region ap-southeast-1 \
--owners 099720109477 \
--filters \
Name=root-device-type,Values=ebs \
Name=architecture,Values=x86_64 \
Name=name,Values='*ubuntu-trusty-14.04*' \
--query 'sort_by(Images, &CreationDate)[-1]'</pre>
<br />
Take note of the <code>"BlockDeviceMappings"</code> array in the output. You should see two ephemeral devices defined:<br />
<br />
<pre>{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
},
{
"DeviceName": "/dev/sdc",
"VirtualName": "ephemeral1"
}</pre>
<br />
Oops... This means that if you're using Canonical AMIs, your instances will not auto-recover.<br />
<br />
My team and I have been unfortunate enough to confirm this a couple of times in the past month. AWS support and <a href="https://forums.aws.amazon.com/thread.jspa?threadID=201388&messageID=656677#656677">this forum post</a> suggest to <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#Using_OverridingAMIBDM">override block device mapping when launching the instance</a>. However, this implies you're using AWS CLI or AWS Console to do so, what if you're lunching your instances into an Auto Scaling Group?<br />
<br />
The alternative is to bake your own AMI without ephemeral block device mappings, which is what this post is about.<br />
<br />
I'm a big fan of HashiCorp tools and have used Packer in the past to build my own VirtualBox/VMware vagrant boxes, so I decided to revisit it as an AMIs baking tool.<br />
<br />
<a href="https://www.packer.io/intro/getting-started/build-image.html">Packer getting started example</a> gave me a good starting point, I just had to make a few small alterations to accommodate my environment and security constrains.<br />
<br />
Here's my packer template: <a href="https://github.com/sshvetsov/packer-ubuntu-rebake/blob/master/ubuntu-14.04-x86_64.json">https://github.com/sshvetsov/packer-ubuntu-rebake/blob/master/ubuntu-14.04-x86_64.json</a><br />
<br />
Notable deviations from original example:<br />
<ol style="text-align: left;">
<li>Remove <code>"access_key"</code> and <code>"secret_key"</code>, these will be read from <code>~/.aws/credentials</code> or passed via environment with <a href="https://github.com/makethunder/awsudo">awsudo</a>.</li>
<li>Replace <code>"source_ami"</code> with <code>"source_ami_filter"</code> to automatically use the latest AMI.</li>
<li>Add <code>"ami_block_device_mappings"</code> to remove ephemeral device mappings with the help of <code>"no_device"</code> property.</li>
<li>Add <code>"ami_description"</code> and <code>"tags"</code>, cause metadata is good.</li>
<li>Explicitly set <code>"vpc_id"</code>, <code>"subnet_id"</code> and <code>"associate_public_ip_address"</code> via external variables file.</li>
<li>Use <code>"force_deregister"</code> and <code>"force_delete_snapshot"</code> to replace existing AMIs with the same name.</li>
</ol>
The rest is simple: configure your AWS credentials, change the variables sections to match your VPC and run:<br />
<br />
<pre>packer build ubuntu-14.04-x86_64.json</pre>
<br />
In my dev environment I run:<br />
<br />
<pre>awsudo -u sa@dev -- packer build -var-file=vars-dev.json ubuntu-14.04-x86_64.json</pre>
<br />
and a couple of minutes later you have a new Ubuntu AMI without those pesky device mappings:<br />
<br />
<pre>amazon-ebs output will be in this color.
==> amazon-ebs: Force Deregister flag found, skipping prevalidating AMI Name
amazon-ebs: Found Image ID: ami-19c87b7a
==> amazon-ebs: Creating temporary keypair: packer_58cec6dc-dff7-c642-cc8f-3e1fecd15019
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing access to port 22 the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
amazon-ebs: Instance ID: i-0587ac0941f65f4f4
==> amazon-ebs: Waiting for instance (i-0587ac0941f65f4f4) to become ready...
==> amazon-ebs: Adding tags to source instance
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Deregistered AMI hvm-ssd/ubuntu-trusty-14.04-x86_64-server, id: ami-2af74a49
==> amazon-ebs: Deleted snapshot: snap-08dc6ce8b2e28c50d
==> amazon-ebs: Creating the AMI: hvm-ssd/ubuntu-trusty-14.04-x86_64-server
amazon-ebs: AMI: ami-13e85570
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Modifying attributes on AMI (ami-13e85570)...
amazon-ebs: Modifying: description
==> amazon-ebs: Modifying attributes on snapshot (snap-01d7b1b146e61d64c)...
==> amazon-ebs: Adding tags to AMI (ami-13e85570)...
==> amazon-ebs: Tagging snapshot: snap-01d7b1b146e61d64c
==> amazon-ebs: Creating AMI tags
==> amazon-ebs: Creating snapshot tags
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.
==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
ap-southeast-1: ami-13e85570</pre>
<br />
Check to confirm with this command (replacing <code>ami-XXXXXXXX</code> with AMI ID returned by packer):<br />
<br />
<pre>aws ec2 describe-images --image-ids ami-XXXXXXXX</pre>
<br />
<b>References</b>:<br />
<ul style="text-align: left;">
<li><a href="https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/">AWS Blog: New – Auto Recovery for Amazon EC2</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html">AWS Docs: Recover Your Instance</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstanceRecovery.html">AWS Docs: Troubleshooting Instance Recovery Failures</a></li>
<li><a href="https://forums.aws.amazon.com/thread.jspa?threadID=201388&messageID=656677#656677">AWS Forums: Can not configure auto-recovery on an eligible instance</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#Using_OverridingAMIBDM">AWS Docs: Updating the Block Device Mapping when Launching an Instance</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html">AWS Docs: Amazon EC2 Instance Store</a></li>
<li><a href="https://www.packer.io/intro/getting-started/build-image.html">Packer: Build an Image</a></li>
</ul>
</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-50630630328834614032015-02-19T17:51:00.003+07:002015-02-19T18:19:06.771+07:00Enable "Developer options" on Android<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipxrxz0T_DTqkbvN-LsS3VcwMs1VqJsItXM4q6zGqTrxqVFRIYZ_TkdZewtfHkR7RUQjb-oVOPCNDjlem6oGhW28HXtGMx52fpFXaL96xDoejxDsmEQjrJb2E0uDu8KgCUzYIxxH4Rz58/s1600/dev-ops.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipxrxz0T_DTqkbvN-LsS3VcwMs1VqJsItXM4q6zGqTrxqVFRIYZ_TkdZewtfHkR7RUQjb-oVOPCNDjlem6oGhW28HXtGMx52fpFXaL96xDoejxDsmEQjrJb2E0uDu8KgCUzYIxxH4Rz58/s1600/dev-ops.png" height="400" width="225" /></a></div>
<div dir="ltr" style="text-align: left;" trbidi="on">
Since release of Android 4.2 the "Developer options" menu is not visible in the Settings by default. Regular users won't miss it, but for those of us who like to tinker with ADB and need the "USB debugging" option enabled, here's how to get it back:<br />
<ol style="text-align: left;">
<li>Go to the "Settings" menu.</li>
<li>Scroll to the bottom and tap "About phone" menu.</li>
<li>Scroll to the bottom and tap on "Build number" seven (7) times.</li>
</ol>
After this your "Developer options" should be back in the "Settings" menu and you be able to start voiding your warrantee like a hacker. "You are now a developer!"</div>
</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-90162430145385252012014-11-29T19:17:00.000+07:002015-10-25T20:10:39.161+07:00Homebrew cheat sheet and workflow<div dir="ltr" style="text-align: left;" trbidi="on"><div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGBKJvAxOmpM14kUaNk6tZ6tvQOu7HTPvZZk0WDwPFikQYr3HH8FI2uwOBqDc3QeGCFPLT8fPGlCqKRZVHku4gzy5ZFFjTeCKdggopHbrsHWsYdrZ8rAMDdt8N5YTab8hYyPw-TDnHTak/s1600/osx-terminal-150.png" /></div>If you are a Mac owner and a serious *NIX user, you're probably aware of <a href="http://brew.sh/">Homebrew</a>.<br />
<br />
I have a few of my favorite tools installed with brew and I like to keep my "Cellar" clean and up-to-date. Here's my commonly used commands and my typical workflow.<br />
<br />
Never forget about man's best friend - "man". Use <tt>man</tt> to get the list of commands and options that can be used with brew:<br />
<br />
<pre>man brew</pre><br />
To list software you currently have installed with version numbers, use:<br />
<br />
<pre>brew list --versions</pre><br />
To see just the software, which is not a dependency of another:<br />
<br />
<pre>brew leaves</pre><br />
To keep the Homebrew itself up-to-date, and fetch the newest version from GitHub:<br />
<br />
<pre>brew update</pre><br />
After updating the brew, check which formulae have an updated version available, display detailed version information to see if you have more than one older version laying around:<br />
<br />
<pre>brew outdated --verbose</pre><br />
See any app that you no longer need and want to get rid of them? Check the dependencies for all installed formulae:<br />
<br />
<pre>brew deps --installed</pre><br />
For even more detailed picture, show the dependencies for all installed formulae as a tree:<br />
<br />
<pre>brew deps --installed --tree</pre><br />
As a final precaution before removing a formula, see what other installed formulae use it as a dependency:<br />
<br />
<pre>brew uses --installed formula</pre><br />
Uninstall formulae and all their older versions:<br />
<br />
<pre>brew remove --force formulae</pre><br />
Upgrade remaining formulae:<br />
<br />
<pre>brew upgrade</pre><br />
Show what will be removed by <tt>cleanup</tt> command, but do not actually remove anything:<br />
<br />
<pre>brew cleanup -ns</pre><br />
Clean the "Cellar" removing any older versions of installed formulae and clearing old downloads from the Homebrew download-cache. Additionally, scrub the cache, removing downloads for even the latest versions of formula, which are downloaded, but not installed:<br />
<br />
<pre>brew cleanup -s</pre></div>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-48968498037451193752014-03-13T21:30:00.000+07:002017-07-02T15:05:01.214+07:00Add time stamp to a file name from CLI<div dir="ltr" style="text-align: left;" trbidi="on"><div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipGKreNDBQlGTJxJBKVH1TrzHzCGQCzYP2FM-e_oTgOoDRh1N9UpcgE8FfdWZ9rYinosdXQif1MvaZ4eSJ84eZvwrCUSvbOCb0XlZbpLBaNEVXou3HO0umTIMbxW7HV_o_c4FJ2LkbbAg/s1600/cli-150x150%255B2%255D" /></div>I often find myself in need to append a time stamp to a file name, but always forget the right command line parameters. Here's a quick reminder to myself on how to do just that.<br />
<br />
<pre>mv filename filename-$(date '+%FT%T')</pre>or<br />
<pre>mv filename filename-$(date '+%s')</pre><br />
For those who want to know the details, here's the break down.<br />
<a name='more'></a><br />
Almost all *nix systems have the <code>date</code> command, which can print or set system date and time. Simply type <code>date</code> and current date and time are displayed:<br />
<br />
<pre>Mon Aug 19 20:37:57 WIT 2013</pre><br />
By default, the date is printed in a "human friendly" form. Unfortunately, it's long, contains spaces, which need escaping and doesn't sort well. A more useful format would look like so: <code>2013-08-19T20:37:57</code>. This format is also known as <a href="http://en.wikipedia.org/wiki/ISO_8601#Combined_date_and_time_representations">ISO 8601</a> standard.<br />
<br />
We can specify the output format by passing it to <code>date</code> command like so:<br />
<br />
<pre>date '+%Y-%m-%dT%H:%M:%S'</pre><br />
Or if you're using a relatively modern *nix distribution like so:<br />
<br />
<pre>date '+%FT%T'</pre><br />
To append current time stamp to a file name, run:<br />
<br />
<pre>mv filename filename-$(date '+%FT%T')</pre><br />
To create a new file with current time stamp in its name, run:<br />
<br />
<pre>touch filename-$(date '+%FT%T')</pre><br />
For the truly hardcore - get <a href="https://en.wikipedia.org/wiki/Unix_time">Unix timestamp</a>, like '1498982531', using:<br />
<br />
<pre>date '+%s'</pre><br />
References:<br />
<ul style="text-align: left;"><li><a href="http://www.manpages.info/linux/date.1.html">http://www.manpages.info/linux/date.1.htm </a></li>
<li><a href="http://www.24hourapps.com/2009/02/linux-tips-8-time-stamp-file-names-for.html">http://www.24hourapps.com/2009/02/linux-tips-8-time-stamp-file-names-for.html</a></li>
</ul></div>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-80458057979237691312013-07-27T19:48:00.004+07:002013-08-19T13:34:12.870+07:00Root Android 4.3 Galaxy Nexus from OS X<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1n6KYd2F8wNiRJOi2fK-b1hInWkLrQUjEllnlB2VNCUL3qyKx1lI82rMv-cCe5OzL-IEise-TURz927NPgTFSoSpx-waqoPKcYTonQsgxzfbsP6qKNvpv16PQo8WIQkxw72YfhZ0p0HM/s1600/galaxy-nexus-root.png" /></div>
My trusty Galaxy Nexus has just received its fresh Android 4.3 OTA, which, on the plus side, brought "improvements to performance and stability", but has also wiped out root along the way. When life gives you lemons... you write a blog post about delicious lemonade you've made, so I decided to update my <a href="http://blog.shvetsov.com/2012/08/rooting-jelly-bean-galaxy-nexus-from-os.html">old guide on how to root JB Galaxy Nexus</a>. Check the link if you need more detailed explanation of each step.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfD9iydVWXvsXPBKi9XAo884uJQsfP3jOUDvy_FI75bkEZiz7GPXIpLY8Vj_WMgDIQQkM_gwGp1RXz-MmHLGC2qLdXj-cQJgbwGjgiRQr2Q9RWcB_-fJtvx9maCmTlC0OKIMLwTZuB1U4/s1600/android-43.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfD9iydVWXvsXPBKi9XAo884uJQsfP3jOUDvy_FI75bkEZiz7GPXIpLY8Vj_WMgDIQQkM_gwGp1RXz-MmHLGC2qLdXj-cQJgbwGjgiRQr2Q9RWcB_-fJtvx9maCmTlC0OKIMLwTZuB1U4/s1600/android-43.png" width="180" /></a></div>
<br />
First thing first, if you haven't done so yet, reveal Developer options in settings, enable USB debugging, <a href="http://blog.shvetsov.com/2012/08/upgrade-galaxy-nexus-to-jelly-bean-with.html">unlock your bootloader</a> and <a href="http://blog.shvetsov.com/2012/08/install-adb-and-fastboot-on-os-x.html">install or update adb, fastboot and other tools</a>. I've written posts about that in the past, so follow the links if you need a walkthrough.<br />
<br />
Download the latest ClockworkMod Recovery image for Galaxy Nexus (GSM) from <a href="http://clockworkmod.com/rommanager">ClockworkMod site</a>, at the moment it's <code>recovery-clockwork-touch-6.0.3.4-maguro.img</code> and save it as <code>cwm.img</code> in your home directory.<br />
<br />
Download the latest SuperSU CWM installable ZIP using the link in the <a href="http://forum.xda-developers.com/showthread.php?t=1538053">XDA-Developers forum thread</a> (<code>UPDATE-SuperSU-v1.45.zip</code> at the time of this writing) and save it in the same directory as CWM Recovery image.<br />
<br />
Make sure <code>adb</code> is detecting your devices correctly and is displaying phone's serial number when you run:<br />
<br />
<pre>adb devices</pre>
<br />
You should see similar output in the terminal:<br />
<br />
<pre>List of devices attached
0149AXXXXXXXXXXXX device</pre>
<br />
Push the SuperSU installer to your sdcard with command:<br />
<br />
<pre>adb push UPDATE-SuperSU-v1.45.zip /sdcard/</pre>
<br />
Reboot into bootloader with command:<br />
<br />
<pre>adb reboot-bootloader</pre>
<br />
Make sure <code>fastboot</code> is also detecting your device and is displaying phone's serial number when you run:<br />
<br />
<pre>fastboot devices</pre>
<br />
Boot into CWM recovery using cwm.img file in your home directory:<br />
<br />
<pre>fastboot boot cwm.img</pre>
<br />
You will see the following output in the terminal:<br />
<br />
<pre>downloading 'boot.img'...
OKAY [ 0.707s]
booting...
OKAY [ 0.388s]
finished. total time: 1.096s</pre>
<br />
Device will boot in to CWM recovery and present you with various options. Choose <tt>install zip</tt> option, then <tt>chose zip from sdcard</tt>, then <tt>0/</tt> - this is the default path to internal sdcard, then find and select the <tt>UPDATE-SuperSU-v1.45.zip</tt> file we pushed earlier. Chose <tt>Yes - Install UPDATE-SuperSU-v1.45.zip</tt> when presented with confirmation dialog and SuperSU will be installed from sdcard.<br />
<br />
Once installation is complete select <tt>+++++Go Back+++++</tt>, then <tt>reboot system now</tt>. I personally choose not to disable recovery flash when presented with the next prompt.<br />
<br />
Once the phone has restarted test for presence of root prompt (<code>#</code>) using:<br />
<br />
<pre>adb shell su</pre>
<br />
You might want to update the <a href="https://play.google.com/store/apps/details?id=eu.chainfire.supersu">SuperSU app</a> via GooglePlay if you're not using the latest version, and it's done.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-71450742623213732172013-04-24T22:30:00.000+07:002013-07-02T17:15:06.647+07:00Using Git with Dropbox<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHvI2cOXAQv6WXDd0oEqIdKwo9HkxMEsN9v2K0I-RgVQlZtJh0DiRyxBFnQBoIBkG8crdP3SqG9Q0IFu7VJXjplxN8dJsIDQ89rpoQ8UFL-GJp6ExeI550Fl8-DpA_texbUGLl5_opLnE/s1600/git-dropbox.png" /></div>
Recently I challenged myself to get much more familiar with <a href="http://git-scm.com/">Git</a>. I've used it briefly in the past, but some of my recent work got me seriously craving for a better version control system (VCS) than Subversion. Something faster, something that didn't require connection to central server, yet still allowed me to access my work on both my home and work computers. Git alone met majority of these requirements, I just needed a remote repository to share and sync my work between computers.<br />
<br />
GitHub was my first choice, but my some of my work is not supposed to be available to the public, and I didn't think premium account was worth it. Setting up my own server seemed like too much work. My search for something simple, free, yet fast and reliable pointed me to <a href="https://www.dropbox.com/referrals/NTY1MDg4MTk">Dropbox</a>.<br />
<br />
Dropbox has always been my favorite way to sync files and passwords between my devices. Here's how to turn it into a perfect remote Git repository.<br />
<br />
First, we need a local Git repository with our work in it. If you don't already have one create one:<br />
<br />
<pre>git init /path/to/new-project</pre>
<br />
It's good to have a <tt>README</tt> file, so let's make one:<br />
<br />
<pre>cd /path/to/new-project
touch README</pre>
<br />
Now, let's stage and commit our changes:<br />
<br />
<pre>git add .
git commit -m "Initial Commit"</pre>
<br />
Next, we create an empty "remote" repository in our Dropbox folder:<br />
<br />
<pre>git init --bare ~/Dropbox/git/new-project.git</pre>
<br />
Technically, it's still stored locally on the same machine, but because it's inside our Dropbox folder, it will be automagically synced to the cloud and other linked devices.<br />
<br />
We still need to "add" our remote repository to the local one:<br />
<br />
<pre>git remote add dropbox ~/Dropbox/git/new-project.git</pre>
<br />
I'm naming my remote repository "dropbox" instead of traditional "origin". This could be handy if you have multiple remote repositories.<br />
<br />
Then, push local master branch to remote:<br />
<br />
<pre>git push -u dropbox master</pre>
<br />
I am also telling git to track remote master branch as upstream with <tt>-u</tt> flag.<br />
<br />
Now I can work in my local repository, make changes and commits, and periodically updating remote repository with:<br />
<br />
<pre>git push</pre>
<br />
If I need to work on my home computer, I can simply clone my project from remote repository inside my Dropbox with:<br />
<br />
<pre>git clone -o dropbox ~/Dropbox/git/new-project.git</pre>
<br />
If local repository already exists on my home computer, but doesn't have the latest changes, we can fetch and merge them from remote with:<br />
<br />
<pre>git pull</pre>
<br />
Now we have a distributed VCS with a central server, which is fast, reliable, super easy to setup, doesn't require maintenance, private and is free of charge. It might not be suitable for collaboration between multiple developers, but it's ideal for the needs of one.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-90634263852321702082013-02-14T22:29:00.001+07:002013-07-28T19:54:51.742+07:00Grab Android screenshot to computer via ADB<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhabglajs3Do448rygw64sYEOIFjrRrci-FsyUnBPZt9EBXy344d2eFs2hEALXs1-fj92XGadWAMp6Fm4r3OYE4TMyI-gv70uj8UyDUPlNeAIBQgaAOVXvJjLaSQ1l-EmGMedjK7P1qvfc/s1600/android-cli-150x150.png" /></div>
There are many ways to take a screen shot on Android device. One simple way to capture the screen on Galaxy Nexus is to simultaneously press and hold <tt>Power</tt> and <tt>Volume Down</tt> buttons. The image will be saved in a "Screenshot" directory and accessible via Gallery.<br />
<br />
Quite often, however, I need to copy the captured image over to my computer. Usually, I do this with <code>adb pull</code> command, but if I'm going to use CLI to retrieve the file, why don't I take the screen shot with it as well?<br />
<br />
One method is to use <code>screencap</code> command via <code>adb shell</code> like so:<br />
<br />
<pre>adb shell screencap -p /sdcard/screen.png
adb pull /sdcard/screen.png
adb shell rm /sdcard/screen.png</pre>
<br />
Not bad, but seems like there's some room for improvement.<br />
<br />
Information provided by <code>screencap -h</code> indicates that screen shot can be sent to <tt>stdout</tt>, but running <code>adb shell screencap -p > screen.png</code> results in seemingly corrupt file.<br />
<br />
Luckily, I wasn't the first person looking into <a href="http://stackoverflow.com/questions/13578416/read-binary-stdout-data-from-adb-shell">this issue</a>. Apparently, <tt>adb shell</tt> is performing an EOL (end-of-line) character conversion, from <tt>LF</tt> (line feed, '<tt>\n</tt>', <tt>0x0A</tt>) to <tt>CR+LF</tt> (carriage return followed by line feed, '<tt>\r\n</tt>', <tt>0x0D0A</tt>).<br />
<br />
If that sounds familiar, that's because <a href="http://blog.shvetsov.com/2012/04/covert-unix-windows-mac-line-endings.html">I've dealt with this before</a>. Unfortunately, this time we're working with binary data, so using <code>tr -d '\r'</code> command to remove all <tt>0x0D</tt> will likely corrupt it by also removing bytes that aren't part of the <tt>0x0D0A</tt> sequence. We want to only remove carriage return characters when they are followed by a line feed.<br />
<br />
The solution is to use <tt>sed</tt> search and replace as follows:<br />
<br />
<pre>adb shell screencap -p | sed 's/\r$//' > screen.png</pre>
<br />
Unfortunately, I found that <tt>sed</tt> fix while works on Ubuntu doesn't work on OS X. This solution using <tt>perl</tt> handles binary search and replace better:<br />
<br />
<pre>adb shell screencap -p | perl -pe 's/\x0D\x0A/\x0A/g' > screen.png</pre>
<br />
Now we have a one-liner that grabs a screen shot from the Android device and stores directly on the computer executing the command. Beautiful.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-59262244472301161802013-02-01T15:06:00.000+07:002014-05-27T17:58:57.418+07:00Access Android app data without root<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhabglajs3Do448rygw64sYEOIFjrRrci-FsyUnBPZt9EBXy344d2eFs2hEALXs1-fj92XGadWAMp6Fm4r3OYE4TMyI-gv70uj8UyDUPlNeAIBQgaAOVXvJjLaSQ1l-EmGMedjK7P1qvfc/s1600/android-cli-150x150.png" /></div>
Recently I recommended a certain data collecting Android app to my coworker. The application was great, but it was designed to upload collected data to the cloud and didn't offer any means to export or backup information locally.<br />
<br />
This isn't an issue on my <a href="http://feedproxy.google.com/~r/blog/random-stuff/~3/QbVqTsyDCXw/rooting-jelly-bean-galaxy-nexus-from-os.html">rooted Galaxy Nexus</a>. Usually, I connect my phone to my computer with USB cable, open CLI and do the following, where <tt>app.package.name</tt> should be replaced with the actual package name of the application:<br />
<br />
<pre>adb shell
su
cp /data/data/app.package.name/databases/application.sqlite /sdcard/
exit
exit
adb pull /sdcard/application.sqlite ~/</pre>
<br />
<b>Update</b>: after I've discovered <a href="http://blog.shvetsov.com/2013/02/grab-android-screenshot-to-computer-via.html">this neat trick</a>, the above can be reduced to one line:<br />
<br />
<pre>adb shell su -c cat /data/data/app.package.name/databases/application.sqlite | sed 's/\r$//' > application.sqlite</pre>
<br />
On my coworker's phone, however, this wouldn't work since he was using a stock, non-rooted Android OS.<br />
<br />
I Googled around for a solution and <i>seemingly</i> found <a href="http://denniskubes.com/2012/09/25/read-android-data-folder-without-rooting/">one</a>. If the application, whose data you wish to access, is debuggable, its read-protected folder can be accessed with the help of the <tt>run-as</tt> command. In essence, we pretend to be the application in question and copy file(s) stored inside the application's data folder into user readable folder like so:<br />
<br />
<pre>adb shell
run-as app.package.name \
cp /data/data/package.name/databases/application.sqlite /sdcard/
exit
adb pull /sdcard/application.sqlite ~/</pre>
<br />
Unfortunately, in this case the application was not debuggable, so the method above did not work either. It appeared that reading Android application data folder with root access was impossible...<br />
<br />
At this point I remembered that starting with Android v4.0 (Ice Cream Sandwich) Google has provided a way to <a href="http://blog.shvetsov.com/2012/09/backup-your-android-without-root-or.html">backup data and applications from Android devices without root</a> via adb. So all I had to do in order to pull that application's data from the device is to run:<br />
<br />
<pre>adb backup -f ~/data.ab -noapk app.package.name</pre>
<br />
This will prompt you to "unlock your device and confirm the backup operation". To keep things simple <i>do not</i> provide a password, otherwise you will have to jump through the hoops to decrypt it later. Just click on "Back up my data" button. The screen will display the name of the package you're backing up, then close by itself upon successful completion.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgPZ5qV881DKwd8MhmpLBTZtRBWx0m7s_WSi0nk2MZliMH24BVpMey2cGZvl18NRSGp_cM2kcAm2uo78s5vrOZqwPcP-1n8O1P_4a-aS0-NxmSpFJVQ58eigIncUAnN4cnUDiyIieHyjw/s1600/backup-screen.png" /></div>
<br />
The resulting "<tt>.ab</tt>" file contains application data in <b>a</b>ndroid <b>b</b>ackup format, which, thanks to <a class="g-profile" href="http://plus.google.com/117221066931981967754" target="_blank">+Nikolay Elenkov</a>, is very well explained and documented in his <a href="http://nelenkov.blogspot.jp/2012/06/unpacking-android-backups.html">excellent blog post</a>. Basically, it's a tar archive that has been run through deflate and optionally encrypted (in a somewhat peculiar way) by <tt>AES-256-CRC</tt> cypher. Nikolay even went as far as to write a <a href="https://github.com/nelenkov/android-backup-extractor">convenient Java program</a> that can pack and unpack such Android backups with and without encryption.<br />
<br />
To quickly extract a simple non-encrypted backup (you did omit the backup password as I suggested, didn't you?) run:<br />
<br />
<pre>dd if=data.ab bs=1 skip=24 | openssl zlib -d | tar -xvf -</pre>
<br />
<b>Update</b>: It was brought to my attention that not all <tt>openssl</tt> installations are compiled with <tt>zlib</tt> support. Here's an alternative one-liner that makes use of python to achieve the same result:<br />
<br />
<pre>dd if=data.ab bs=1 skip=24 | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" | tar -xvf -</pre>
<br />
The result is the <tt>apps/app.package.name/</tt> folder containing application data, such as SQLite database I was particularly interested in and the application preferences.<br />
<br />
Enjoy!<br />
<br />
<b>Update</b>: A few people mentioned in comments below and on <a href="https://stackoverflow.com/questions/9997976/android-pulling-sqlite-database-android-device/14686392#14686392">StackOverflow</a> that this method doesn't work if application developer has explicitly disabled ability to backup his app by setting <tt>android:allowBackup="false"</tt> in the application manifest.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-87023232082273317772012-09-04T16:26:00.002+07:002013-02-14T17:39:17.019+07:00Backup your Android without root or custom recovery<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhabglajs3Do448rygw64sYEOIFjrRrci-FsyUnBPZt9EBXy344d2eFs2hEALXs1-fj92XGadWAMp6Fm4r3OYE4TMyI-gv70uj8UyDUPlNeAIBQgaAOVXvJjLaSQ1l-EmGMedjK7P1qvfc/s1600/android-cli-150x150.png" /></div>
Recently discovered a neat new way to back up apps on my Android without having to use Titanium Backup, having to unlock bootloader or root the device to take a NANDroid snapshot. The icing on the cake - I can do it from command line!<br />
<br />
<b>Warning</b>: this method is only available to devices running Ice Cream Sandwich or higher and <a href="http://code.google.com/p/android/issues/detail?id=28303">it has bugs</a>, but it is an alternative for folks who want to make a backup <i>before</i> <a href="http://feedproxy.google.com/~r/blog/random-stuff/~3/QbVqTsyDCXw/rooting-jelly-bean-galaxy-nexus-from-os.html">rooting their device</a>.<br />
<br />
You'll need to have the <a href="http://blog.shvetsov.com/2012/08/install-adb-and-fastboot-on-os-x.html">latest adb installed and in you path</a>, USB debugging enabled and your phone connected to your Mac.<br />
<br />
The command in question has the following format:<br />
<br />
<code>adb backup [-f <file>] [-apk|-noapk] [-shared|-noshared] [-all] [-system|-nosystem] [<packages...>]</code><br />
<br />
Here are some common uses:<br />
<br />
1. Backup all non-system apps, their data and your shared data (i.e. SD card contents):<br />
<br />
<pre>adb backup -f /path/to/backup-file -apk -shared -all -nosystem</pre>
<br />
2. Backup all apps, their data and shared data (note: won't work on DRM protected apps):<br />
<br />
<pre>adb backup -f /path/to/backup-file -apk -shared -all -system</pre>
<br />
3. Backup only app data (not the APKs themselves) and your shared data:<br />
<br />
<pre>adb backup -f /path/to/backup-file -all</pre>
<br />
4. Backup only non-system apps:<br />
<br />
<pre>adb backup -f /path/to/backup-file -apk -noshared -nosystem</pre>
<br />
5. Backup only your shared data:<br />
<br />
<pre>adb backup -f /path/to/backup-file -noapk -shared -nosystem</pre>
<br />
6. Restore everything from your backup:<br />
<br />
<pre>adb restore /path/to/backup-file</pre>
<br />
During backup your phone will display a backup confirmation screen giving you an option to enter an encryption password. In my case it would only work with non-empty password, otherwise it would generate a zero-sized file.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-35361744212425175212012-08-19T19:03:00.001+07:002013-02-14T14:31:19.950+07:00Install adb and fastboot on OS X<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipGKreNDBQlGTJxJBKVH1TrzHzCGQCzYP2FM-e_oTgOoDRh1N9UpcgE8FfdWZ9rYinosdXQif1MvaZ4eSJ84eZvwrCUSvbOCb0XlZbpLBaNEVXou3HO0umTIMbxW7HV_o_c4FJ2LkbbAg/s1600/cli-150x150%255B2%255D" /></div>
There are plenty of packages floating around the Internet with just the <code>adb</code> and <code>fastboot</code> utilities for OS X, but I prefer installing those (and few other android utilities) on OS X is via Android SDK. This way you know exactly what you're getting and it's easy to can keep them up-to-date with Android SDK Manager.<br />
<br />
First download the latest Android SDK package from <a href="http://developer.android.com/sdk/index.html" rel="nofollow">http://developer.android.com/sdk/index.html</a> (<code>android-sdk_r20.0.3-macosx.zip</code> at the time of this writing).<br />
<br />
Change into your download directory and unzip archive into your preferred location. I like my home directory so I run:<br />
<br />
<pre>unzip android-sdk_r20.0.3-macosx.zip -d ~</pre>
<br />
This creates a <code>android-sdk-macosx</code> directory in our home directory. Now run <code>android</code> application inside the <code>tools</code> directory to open Android SDK Manager (ASM).<br />
<br />
<pre>~/android-sdk-macosx/tools/android</pre>
<br />
ASM will connect to various repositories and present you with all possible Tools, Platforms and Add-Ons that could be downloaded and installed. Since we're only interested in few specific tools, deselect all except "Android SDK Tools" and "Android SDK Platform-tools" packages and then install / upgrade them.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpl3fqFzpSVRDqEd2GxzF1S26G7op7R4lthgPQQRvw6tbe21gJSaz5n6HDE2BucCnzGJP0p61h_B3tBcKDF5o3Ul63KAhYwLNhGjyF6pbPb2Mst8UgNyCs5onmWXG0O5-zV5Ae-SsxjO0/s1600/android-sdk-manager.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpl3fqFzpSVRDqEd2GxzF1S26G7op7R4lthgPQQRvw6tbe21gJSaz5n6HDE2BucCnzGJP0p61h_B3tBcKDF5o3Ul63KAhYwLNhGjyF6pbPb2Mst8UgNyCs5onmWXG0O5-zV5Ae-SsxjO0/s400/android-sdk-manager.jpg" width="400" /></a></div>
<br />
Note: to accelerate checking for updates during future runs, go to "Tools"->"Manage Add-on Sites..." and click on "Disable All" button in "Official Add-on Sites" tab. From now on ASM will check only the default repository containing tools and APIs.<br />
<br />
Now we'll add android tools to our PATH to make executing them more convenient. Create or edit <code>.bash_profile</code> file in your home directory to contain these lines, just don't forget to replace <code>YOURUSERNAME</code> with your actual user name:<br />
<br />
<pre>ANDROID_TOOLS=/Users/YOURUSERNAME/android-sdk-macosx
export PATH=${PATH}:${ANDROID_TOOLS}/platform-tools:${ANDROID_TOOLS}/tools</pre>
<br />
Done.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-22734788637120097802012-08-15T15:22:00.001+07:002013-02-14T14:31:19.947+07:00Rooting Jelly Bean Galaxy Nexus from OS X<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1n6KYd2F8wNiRJOi2fK-b1hInWkLrQUjEllnlB2VNCUL3qyKx1lI82rMv-cCe5OzL-IEise-TURz927NPgTFSoSpx-waqoPKcYTonQsgxzfbsP6qKNvpv16PQo8WIQkxw72YfhZ0p0HM/s1600/galaxy-nexus-root.png" /></div>
Now that we've <a href="http://blog.shvetsov.com/2012/08/upgrade-galaxy-nexus-to-jelly-bean-with.html">unlocked bootloader on our Galaxy Nexus and updated OS to Jelly Bean 4.1.1</a>, rooting it is extremely simple. In fact, to root GNex all you need is <code>su</code> (Super User) binary as one of the system apps. Since <code>/system</code> partition is mounted as Read Only, we need to boot into custom recovery mode and install <code>su</code> utility on our phone from there.<br />
<br />
In this example we'll use ClockworkMod Recovery. You don't have to install the ClockworkMod Rom Manager application, or replace your stock recovery partition with CWM's. Having unlocked bootloader allows us to boot into CWM Recovery without overwriting anything on our device, as you would boot into Linux on PC from a LiveCD/USB.<br />
<a name='more'></a><br />
<br />
Download a latest CM Recovery image for Galaxy Nexus from <a href="http://clockworkmod.com/rommanager">ClockworkMod site</a> (<code>recovery-clockwork-6.0.1.0-maguro.img</code> at the time of this writing), whether it is Touch-capable, i.e. allows navigation using touch screen instead of Volume buttons, is up to you. Place it into your home directory and rename it into something short and simple like <code>cwm.img</code>.<br />
<br />
Now we need a <code>su</code> binary. There are two popular options at the moment: <a href="https://play.google.com/store/apps/details?id=eu.chainfire.supersu" rel="nofollow">SuperSu</a> and <a href="https://play.google.com/store/apps/details?id=com.noshufou.android.su" rel="nofollow">Superuser</a>. Pick the one you like best. I'm using SuperSu, so I've downloaded the latest CWM installable archive from <a href="http://forum.xda-developers.com/showthread.php?t=1538053">this thread</a> (<code>CWM-SuperSU-v0.94.zip</code> at the time of this writing) and placed it in my home directory.<br />
<br />
Now we need to move su-installer into our phone. As usual, I prefer the CLI approach: connect your phone with USB cable (make sure USB debugging is enabled) and run:<br />
<br />
<pre>adb push CWM-SuperSU-v0.94.zip /sdcard/</pre>
<br />
With su-installer in place, we need to reboot our phone into fastboot mode. Either shut it down and start it by holding Volume Up + Volume Down + Power Button, or simply do it from command line:<br />
<br />
<pre>adb reboot-bootloader</pre>
<br />
Once our phone in fastboot mode (screen showing a green droid laying on its back with its chest plate open), test whether it's recognized by running:<br />
<br />
<pre>fastboot devices</pre>
<br />
If it is, you'll see your device's serial number followed by word "fastboot". We can now boot into custom recovery we've downloaded earlier by running:<br />
<br />
<pre>fastboot boot cwm.img</pre>
<br />
This will restart our phone and boot it into ClockworkMod Recovery.<br />
<br />
Once in recovery mode, choose <code>install zip from sdcard</code>, then <code>choose zip from sdcard</code> and select the <code>CWM-SuperSU-v0.94.zip</code> file we've uploaded earlier.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlhdS_meW8yog0t2Q7Payunr-0VB13UShj0EHTNdk5plaO-nrfF7f_a7GPmCsi5rebPlP8vQnOlDmCcP2sTg4ttOkOJhrcpJfISxkcFKW9Sp2rOnH26klPIFvxq9SDkFriHCFV6Mcg-n4/s1600/install-su.jpg" /></div>
<br />
After installation is complete we can reboot and enjoy our freshly rooted Galaxy Nexus.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-16149067517844328482012-08-08T14:28:00.001+07:002013-02-14T14:31:19.944+07:00Upgrade Galaxy Nexus to Jelly Bean with fastboot on OS X.<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhduF2a7ThaY9ASUlfBjNUJOuXDWmyle_XkveFxZv2GJmUz5ZrMUQARsf_bCH1yYkWET5Fw1pwt3UWqmN6XEW9fQc2i7GEqcVa5Oxo6nDjd3qIruDh5xJfnOUrIT6IeaOkbjBa0ZUdwI6k/s1600/galaxy-nexus-jb.png" /></div>
Great news for Galaxy Nexus owners that are still waiting for their Over The Air (OTA) update to Jelly Bean. Last week Google released Jelly Bean 4.1.1 factory images for most Nexus devices (no image for CDMA/LTE Galaxy Nexus yet). This is particularly useful for folks that didn't buy their GNex from Play Store (i.e. the <i>yakju</i> / <i>takju</i> variant) and have one of the regional/carrier variants (like <i>yakjuxw</i>, <i>yakjuxs</i>, <i>yakjuux</i>, <i>yakjusc</i>, <i>yakjuzs</i>, <i>yakjudv</i>, <i>yakjukr</i> and <i>yakjujp</i>), which will take even longer to get an OTA update.<br />
<br />
The only requirements are: GSM version of Galaxy Nexus, factory image and fastboot utility in your PATH (I'll make a separate post on <a href="http://blog.shvetsov.com/2012/08/install-adb-and-fastboot-on-os-x.html">how to setup fastboot utility on Mac OS X</a>).<br />
<a name='more'></a><br />
First step is... <b><a href="http://blog.shvetsov.com/2012/09/backup-your-android-without-root-or.html">backup your phone</a></b>. The described procedure will wipe your device clean of all apps, settings and personal data, so do a thorough backup! You've been warned.<br />
<br />
Download the latest <i>yakju</i> or <i>takju</i> image from <a href="https://developers.google.com/android/nexus/images" rel="nofollow">Factory Images for Nexus Devices</a> page. In this example I will use <a href="https://developers.google.com/android/nexus/images#yakju" rel="nofollow">yakju image for GSM Galaxy Nexus</a>, so the file name is <code>yakju-jro03c-factory-3174c1e5.tgz</code><br />
<br />
Untar the image using either <a href="http://wakaba.c3.cx/s/apps/unarchiver.html">Unarchiver</a> or command line:<br />
<br />
<pre>tar zxvf yakju-jro03c-factory-3174c1e5.tgz</pre>
<br />
This creates a new directory <code>yakju-jro03c</code> with a few image files and the <code>flash-all.sh</code> script. Go to command line and change into this new directory.<br />
<br />
Switch GNex into fastboot mode by shutting it down first, then holding: <b>Volume Up + Volume Down + Power button</b>. Your phone should show a green droid lying on his back with his chest plate open. From here you have a few basic options (Start, Restart bootloader, Recovery mode, Power off) that could be selected with volume keys and activated with power key.<br />
<br />
Connect your phone to your Mac with USB cable.<br />
<br />
To see whether device is properly recognized, from command line run:<br />
<br />
<pre>fastboot devices</pre>
<br />
If everything is in order, you'll see something like this:<br />
<br />
<pre>DEVICESERIALNUMBER fastboot</pre>
<br />
We now need to unlock our bootloader in order to flash new image. Run:<br />
<br />
<pre>fastboot oem unlock</pre>
<br />
You will get a warning that this step will completely wipe your phone. Accept it on the phone (volume keys to select and power key to confirm) and your bootloader will be unlocked:<br />
<br />
<pre>...
OKAY [ 18.009s]
finished. total time: 18.009s</pre>
<br />
Now we flash the Jelly Bean by running the aforementioned script:<br />
<br />
<pre>./flash-all.sh</pre>
<br />
The script will begin installing new bootloader, baseband firmware and operating system (once again all user data will be erased).<br />
<br />
<pre>sending 'bootloader' (2308 KB)...
OKAY [ 0.252s]
writing 'bootloader'...
OKAY [ 0.289s]
finished. total time: 0.541s
rebooting into bootloader...
OKAY [ 0.007s]
finished. total time: 0.007s
sending 'radio' (12288 KB)...
OKAY [ 1.334s]
writing 'radio'...
OKAY [ 1.375s]
finished. total time: 2.709s
rebooting into bootloader...
OKAY [ 0.007s]
finished. total time: 0.007s
archive does not contain 'boot.sig'
archive does not contain 'recovery.sig'
archive does not contain 'system.sig'
--------------------------------------------
Bootloader Version...: PRIMELC03
Baseband Version.....: I9250XXLF1
Serial Number........: DEVICESERIALNUMBER
--------------------------------------------
checking product...
OKAY [ 0.007s]
checking version-bootloader...
OKAY [ 0.008s]
checking version-baseband...
OKAY [ 0.008s]
sending 'boot' (4366 KB)...
OKAY [ 0.473s]
writing 'boot'...
OKAY [ 0.268s]
sending 'recovery' (4708 KB)...
OKAY [ 0.513s]
writing 'recovery'...
OKAY [ 0.314s]
sending 'system' (396675 KB)...
OKAY [ 42.561s]
writing 'system'...
OKAY [ 36.033s]
erasing 'userdata'...
OKAY [ 0.290s]
formatting 'userdata' partition...
Creating filesystem with parameters:
Size: 14539534336
Block size: 4096
Blocks per group: 32768
Inodes per group: 8144
Inode size: 256
Journal blocks: 32768
Label:
Blocks: 3549691
Block groups: 109
Reserved block group size: 871
Created filesystem with 11/887696 inodes and 97200/3549691 blocks
sending 'userdata' (137559 KB)...
writing 'userdata'...
OKAY [ 28.331s]
erasing 'cache'...
OKAY [ 0.013s]
formatting 'cache' partition...
Creating filesystem with parameters:
Size: 452984832
Block size: 4096
Blocks per group: 32768
Inodes per group: 6912
Inode size: 256
Journal blocks: 1728
Label:
Blocks: 110592
Block groups: 4
Reserved block group size: 31
Created filesystem with 11/27648 inodes and 3566/110592 blocks
sending 'cache' (8832 KB)...
writing 'cache'...
OKAY [ 3.621s]
rebooting...
finished. total time: 112.474s</pre>
<br />
At this point Google recommends to lock the bootloader with <code>fastboot oem lock</code> command, but if you're planning to <a href="http://blog.shvetsov.com/2012/08/rooting-jelly-bean-galaxy-nexus-from-os.html">root your phone</a> in the near future, leave it unlocked. Locking bootloader means your phone will be wiped next time you unlock it, so unless you planning to return your phone to the store - don't.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-56972086763331589332012-08-05T14:44:00.000+07:002012-08-05T14:44:04.983+07:00Google Apps email, GoDaddy hosting and SPF records<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvC3QUm3VzG54k-gzGQhvcfb0I0sEwwHKWyIKCTVRGCifrw70wqPal8QJqPRO3SMxrbPuHL2RpRofD3aKZhhZRjqSQ9Lruz_Bm1Gv9etzMAsiZfz9wp0D2yGSG8PORJmZdoKP5IeAXXPo/s1600/spf-logo-150.png" /></div>
The other day I was setting up another web site using GoDaddy for hosting and DNS and Google Apps for email. The two Gs make a great combo since GoDaddy allows for quick domain verification right within Google Apps admin view, while their <a href="https://www.godaddy.com/gdshop/google/gmail_login.asp" rel="nofollow">configuration tool</a> automatically creates the necessary MX records for Google Apps. The deployment is as fast as it gets.<br />
<br />
The issue arose when I tried to send test email from the web application using Google's SMTP servers. Apparently GoDaddy rejects all outgoing SMTP connections to any servers but its own <code>relay-hosting.secureserver.net</code>. Not much can be done about that, so I guess I just have to configure my <a href="http://en.wikipedia.org/wiki/Sender_Policy_Framework" rel="nofollow">SPF records</a> to define which servers are authorized to send emails on behalf of my domain.<br />
<br />
For Google Apps it's a well documented <a href="http://support.google.com/a/bin/answer.py?hl=en&answer=178723" rel="nofollow">procedure</a>, just add the following TXT record:<br />
<br />
<code>"v=spf1 include:_spf.google.com ~all"</code><br />
<br />
Now what settings to add for GoDaddy's servers? GoDaddy's SPF wizard would have me use something like this:<br />
<br />
<code>"v=spf1 a mx include:smtp.secureserver.net ~all"</code><br />
<br />
Since I've changed my MX servers to Google's and they won't be sending any mail, the "mx" mechanism could be dropped. Same goes for "a" mechanism since, as I discovered, GoDaddy does not allow its shared hosts to send any email directly. Let's dig into the "include:smtp.secureserver.net" portion and see what's inside...<br />
<br />
<pre>dig smtp.secureserver.net -t txt</pre>
<br />
Here's the contents of TXT record: <code>"v=spf1 include:spf.secureserver.net -all"</code>. OK, let's dig deeper...<br />
<br />
<pre>dig spf.secureserver.net -t txt</pre>
<br />
Now we get a whole set: <code>"v=spf1 include:in.spf.secureserver.net include:in2.spf.secureserver.net include:ext1.spf.secureserver.net include:ext2.spf.secureserver.net include:ext3.spf.secureserver.net include:mon.spf.secureserver.net include:exch.spf.secureserver.net -all"</code>. Yikes! Do I have to check all of these one by one?<br />
<br />
Luckily, there's an easier way with a free online tool called <a href="http://www.unifiedemail.net/Tools/SPFParser/">SPF Parser</a>. Just paste <code>smtp.secureserver.net</code> into the text field and get the entire list of all A and relevant SPF records recursively parsed.<br />
<br />
The output contained IP address of GoDaddy's relay server, so in theory it should work, so my final SPF record becomes:<br />
<br />
<code>"v=spf1 include:_spf.google.com include:smtp.secureserver.net ~all"</code><br />
<br />
Once new record is in place, validate it over at <a href="http://www.kitterman.com/spf/validate.html">Scott Kitterman's site</a>, where you can see what your domain is currently reporting, whether it's valid and whether an email sent from certain IP shall pass of fail SPF verification.</div>
testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-78865993666186665342012-04-18T12:09:00.000+07:002012-04-18T23:04:53.764+07:00Multiple time zones in Google Calendar<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGE8AzDmzBGY_Zs5MRqwtkwMMtcRBhvoRBOK26MoCFKHquXv5TH5-UDga4WUG6xPR46UmFzkEhfvHqqihIg_pqzCG5_P7IuIfx4gm8TE0R9d0Kcx8rmpKri1Ydem__xjKbbyYbJK1oMBY/s1600/gcal.png" /></div>
Imagine this: you're in Jakarta when client from Kuala Lumpur requests a conference call at 3 pm. You start thinking: "Is it 3 pm their time or yours? Must be theirs. What was it again in local time? I know there's an hour difference, but are they ahead of me or behind? I think they're one hour ahead, but better double check..." If you work with clients in different timezones you probably familiar with that scenario.<br />
<br />
Luckily, Google Calendar takes care of this problem quite nicely:<br />
<ol>
<li>Go to Settings -> General -> Your current time zone</li>
<li>Set your primary time zone if you haven't done so yet</li>
<li>Check "Display all time zones" for full list of time zones</li>
<li>Click on the "Show an additional time zone" link to add another time zone<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="55" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx14W9ibS5rgiFNqLh6dnM8bZuwfXQAmp6NQ5LQepZ8aumxxWYmQRQtYi1DDS0xO8tyVmWI-ztIMUCfePkFjhPCnyrDGqVmdtg_T5WmtxraI3xuOLVgWra-ntXBbjJ-mAcJYDauFaHxK0/s400/google-cal-1.jpg" width="400" /></div>
<br />
</li>
<li>Chose you additional time zone from the list</li>
<li>Enter your custom labels<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="61" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX6NJHWir_IxO9fogZImZTNS5ABH9Pz0EtRa5VVTCHSezI7aXdV9Q4htjF-_02za8oT5wSF9OMXjoBzBfyMSzc2fMcUFCdbM_kfFOhvxX_PxevG3EwmItkgTGm9MWGdKx1k43qYZjE5qI/s400/google-cal-2.jpg" width="400" /></div>
<br />
</li>
<li>Save changes, and your appointments will have a nice label for each time zone next to them:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8UhpR066O0P0ugniFZ9hGudF4bEsp19Ju1DKxZGjiPpRBUA9WNxGAKfSHy-JeSqtmJePk2vSQqVbpu0lDNcgQN1wcQ6hKLdnnXWBY4vtxwmL4hHl75IbPWhaUliPCRGO0FlMc8eMbFDg/s1600/google-cal-3.jpg" /></div>
</li>
</ol>
<br />testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-51311046617497815142012-04-16T21:04:00.000+07:002012-04-17T10:35:40.316+07:00Convert Unix, Windows, Mac line endings using OS X command<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipGKreNDBQlGTJxJBKVH1TrzHzCGQCzYP2FM-e_oTgOoDRh1N9UpcgE8FfdWZ9rYinosdXQif1MvaZ4eSJ84eZvwrCUSvbOCb0XlZbpLBaNEVXou3HO0umTIMbxW7HV_o_c4FJ2LkbbAg/s1600/cli-150x150%255B2%255D" /></div>
Today I had to copy some MySQL data from Debian server into test environment on my MacBook. While importing data from tab delimited text files, I noticed warnings that data in the last column of several tables was being truncated. I looked at the tables and noticed MySQL doing some very strange formatting when printing them. It looked almost as if last column was padded with a bunch of white space. I opened import file in <a href="http://www.barebones.com/products/textwrangler/">TextWrangler</a> and it appeared fine, but when I looked in document options, I saw this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNCaIafgkXluo2UchEcoUm8kdYp4OJPqB8N49wZVVM-28EyC8trwAA9svBD4qI0eXFg45TLiWoJqPX_33ewfQR3sxi2x_XgYmivysR3O3hYYkoJ19xOpNgXPiZNGQ5_6FUSQ7tUwcE3gY/s1600/doc-options.jpg" /> </div>
<br />
The good ol' EOL (end-of-line) character...<br />
<br />
Different operating systems use different characters to mark the end of line:<br />
<ul>
<li>Unix / Linux / OS X uses <tt>LF</tt> (line feed, '<tt>\n</tt>', <tt>0x0A</tt>)</li>
<li>Macs prior to OS X use <tt>CR</tt> (carriage return, '<tt>\r</tt>', <tt>0x0D</tt>)</li>
<li>Windows / DOS uses <tt>CR+LF</tt> (carriage return followed by line feed, '<tt>\r\n</tt>', <tt>0x0D0A</tt>)</li>
</ul>
I'm guessing the person who sent me those files first transferred them to his Windows machine in ASCII mode, so newline characters got automatically converted during transfer.<br />
<br />
Since some of the files were very big, instead of changing line endings in TextWrangler I decided to use command line (shocking, I know).<br />
<br />
First I executed<br />
<pre>cat -v file-name
</pre>
to confirm existence of the dreaded <tt>^M</tt> (carriage return) at the end of every line, and then ran<br />
<pre>tr -d '\r' < file-name > file-name-unix
</pre>
to generate new files without <tt>CR</tt> characters.<br />
<br />
<tt>tr</tt> (translate character) is a nice little utility that does just that, substitutes one character with another or deletes it (like in my example). It's available on pretty much any *nix distro so no need to install additional software.testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-49337710092412679082012-04-14T15:43:00.000+07:002012-04-14T15:43:52.243+07:00Mounting CIFS shares from Windows command line<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnAlivnRP1Mo01XCawE8HX3kP3V-NM_gr7_570g8B4i0fX4bS3zb2EKDasrHZe9Ef9VBAWucafz7kOJigCPlDcNWkjyWCfny5AyinjfNzMgK5bMg3Zg-YiTZDFsEaGFDHPPv-Tu51Nh8k/s1600/windows-cli-150x150.png" /></div>
Unfortunately, sometimes even Linux geeks have to do it...<br />
<br />
If you need to mount or unmount ('map' or 'disconnect' in Windows lingo) a Windows (Samba) share from Windows command line or batch file, you can do it with '<tt>net use</tt>' command.<br />
<br />
To map a network drive:<br />
<ol>
<li>Click <b>Start</b>, and then click <b>Run</b> (or use Windows+R keyboard shortcut).</li>
<li>In the <b>Open</b> box, type <tt>cmd</tt> to open command line window.</li>
<li>Type the following, replacing <i>X:</i> with drive letter you want to assign to the shared resource:<br />
<pre>net use X: \\computer_name\share_name</pre>
</li>
</ol>
To disconnect a mapped drive:<br />
<ol>
<li>Open command line window.</li>
<li>Type the following, replacing <i>X:</i> with drive letter of the shared resource:<br />
<pre>net use X: /delete</pre>
</li>
</ol>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-3902503194847212162012-04-07T12:12:00.000+07:002012-04-14T15:38:58.121+07:00How to enable Windows 7 Administrator<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnAlivnRP1Mo01XCawE8HX3kP3V-NM_gr7_570g8B4i0fX4bS3zb2EKDasrHZe9Ef9VBAWucafz7kOJigCPlDcNWkjyWCfny5AyinjfNzMgK5bMg3Zg-YiTZDFsEaGFDHPPv-Tu51Nh8k/s1600/windows-cli-150x150.png" /></div>
The "Administrator" user account is disabled by default in Windows 7. Since I had to enable it a few times for office setups and couldn't recall how (I try to use Windows as little as possible), here's the command:<br />
<br />
<pre>net user administrator /active:yes</pre>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-57862793638808347882011-11-16T19:59:00.001+07:002011-11-16T21:23:28.675+07:00SSH into ESXi 5 host using public key<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmmxRV4DlDcHQfLdMUy2PuSHSjk3ikDZGaIPUkNN8V4Rn_opCmvD1Q99wOGZCYuL80mQDFKzgXuoNPtUuf0mzmAcVFQFsPhmicS0zLsLfZQT5ujSbd98DzfxMZCOi2CSi18OiLNG9DA4g/s1600/vmware-cli-150.png" /></div>
Just a little follow up on my <a href="http://blog.shvetsov.com/2010/06/openssh-public-key-authentication.html">previous post</a>.<br />
<br />
If you want to enable OpenSSH Public Key Authentication on your ESXi 5 host, append contents of your public key to <code>/etc/ssh/keys-USERNAME/authorized_keys</code> file instead of usual <code>~/.ssh/authorized_keys</code> file.<br />
<br />
For example, to enable public key logins as <code>root</code> user, do the following:<br />
<pre>cat ~/.ssh/id_rsa.pub | ssh root@esxi.machine.com 'cat >> /etc/ssh/keys-root/authorized_keys'</pre>
<br />
Obviously, SSH server needs to be enabled on the ESXi host before you can do that. As I mentioned in my <a href="http://blog.shvetsov.com/2011/11/installing-esxi-5-on-intel-dx58so2.html">ESXi 5 installation write up</a>, starting with <strike>vSphere 5</strike> vSphere 4.1 you can easily enable SSH via ESXi's Direct Console User Interface (DCUI) or vSphere Client.testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-17588072759357667692011-11-11T14:07:00.000+07:002012-04-14T16:08:39.747+07:00OpenSSH Public Key Authentication<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGs9Kn7uqzjU0Eo_4aZNuKahBOOdJud-p5nU-jiKHytaW51g0tOh79A7UUSyfWHi3B7sMSdUdREQPY27HQ7x_nPTTbPIVpoJD5oL0Ir0ca56KR9FT_wM5k6VQuNVdOhC1qkyBQS0ywvQw/s1600/openssh-150.png" /></div>
Secure SHell is awesome. I think it is one of the most valuable, yet taken for granted tools in Sysadmin's arsenal. This post is about Public Key Authentication, a feature that makes remote access of multiple systems not only faster, but also more secure.<br />
<br />
I won't talk about how public keys work. If you don't know it yet, you could learn more <a href="http://en.wikipedia.org/wiki/Public-key_cryptography">here</a>. The point is, I use Public Key Authentication (PKA) very often, on all types of OS, to connect to a large variety of hosts. So here are my preferred ways to set it up.<br />
<a name='more'></a><br />
<b>On Unix / Linux / Mac OS X with OpenSSH</b><br />
<br />
Generate your key pair with:<br />
<br />
<pre>ssh-keygen -t rsa -b 4096 -C "sergei@MacBookPro"</pre>
<br />
Argument <code>-t</code> specifies the type of key you're creating (<a href="http://the.earth.li/%7Esgtatham/putty/0.61/htmldoc/Chapter8.html#puttygen-keytype">go with RSA</a>), <code>-b</code> is the encryption level (I'm doubling the default value here), <code>-C</code> adds a comment to the public key (now it's easy to ID my key on the remote host).<br />
<br />
After some time you'll be prompted for keys name / location and the pass<b>phrase</b>. Make the phrase a long one if you want good security, it stops bad people from using your key. A sentence or two from a song / poem / story you like, with punctuation and capitalization will make it more secure, and more memorable, than randomly generated password.<br />
<br />
In some special cases you can just hit "Enter" for password-less key. Less secure, but very useful for some automated tasks.<br />
<br />
By default, private and public keys <code>id_rsa</code> and <code>id_rsa.pub</code> respectively, should now be inside <code>.ssh</code> directory in your current user's home directory.<br />
<br />
Append contents of your public key into the <code>authorized_keys</code> file in the remote host's <code>.ssh</code> directory with either:<br />
<br />
<pre>cat ~/.ssh/id_rsa.pub | ssh user@remote.machine.com 'cat >> .ssh/authorized_keys'</pre>
<br />
or, if <code>ssh-copy-id</code> is available on your system, with:<br />
<br />
<pre>ssh-copy-id user@remote.machine.com</pre>
<br />
<b>On Windows with PuTTY </b><br />
<br />
<a href="http://the.earth.li/%7Esgtatham/putty/latest/x86/puttygen.exe"><code>puttygen.exe</code></a> utility from the excellent <a href="http://www.chiark.greenend.org.uk/%7Esgtatham/putty/">PuTTY suite</a> will allow you to generate private and public keys <code>keyname.ppk</code> and <code>keyname.pub</code>. Note: PuTTY saves public key in SSH-2 format, but it will also give you the OpenSSH format version. Run the utility and do the following:<br />
<ol>
<li>select "SSH-2 RSA" as type of key to generate</li>
<li>enter 4096 in the "Number of bits in a generate key field"</li>
<li>click "Generate" button</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqYOLnJlxVP2Zxxwdy8HsDWFvF4qG5zCGADcnf7PgA7ZFBxEpUsC3AWm75UB5DwOy1TV1wq5mfZ8HccNdydl6qNAHrytmsPqNIJkgbe4jKk7tx-OLOa7KqLSdsnA3x9Zn1-0xA6oVv80k/s1600/puttygen-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="307" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqYOLnJlxVP2Zxxwdy8HsDWFvF4qG5zCGADcnf7PgA7ZFBxEpUsC3AWm75UB5DwOy1TV1wq5mfZ8HccNdydl6qNAHrytmsPqNIJkgbe4jKk7tx-OLOa7KqLSdsnA3x9Zn1-0xA6oVv80k/s320/puttygen-1.png" width="320" /></a></div>
<br />
PuTTYgen will start generating your key. Move your mouse in random manner over the blank area until the bar is full.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr7_0jRC7mmX1KFhXQwKcQZQFEMj9hzssTAE8Sb5-O9R5fPHAHbOl-_4QvhoXEQyXanaeR_pPIua1ghprSpRTzGTpaFCk2YojznP12ip-XvMe4bpL-FaRBy-7qRniXgPpqws6x6WuxFhg/s1600/puttygen-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="307" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr7_0jRC7mmX1KFhXQwKcQZQFEMj9hzssTAE8Sb5-O9R5fPHAHbOl-_4QvhoXEQyXanaeR_pPIua1ghprSpRTzGTpaFCk2YojznP12ip-XvMe4bpL-FaRBy-7qRniXgPpqws6x6WuxFhg/s320/puttygen-2.png" width="320" /></a></div>
<br />
Now to finalize your key:<br />
<ol>
<li>enter your key comment</li>
<li>enter your strong passphrase (or leave it empty if required)</li>
<li>save a copy of your private and/or public key (in SSH-2 format) to some secure location</li>
<li>Right click, "Select all" and copy your public key (in OpenSSH format) to append it to <code>authorised_keys</code> file on remote host</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5VJjHi56UBGhJXsS0z4GDvXtq8Z9nyBBGdI1CQ24yNN5MAKEw5b8yhOhbdk4EVxSL_o5WkLEd-wRNgUMTr7Sxb4pY9UceCEfGl1zAMdoHfQjvW7QZsVzC9JpGe55LaqebJdrPpTn9Bwc/s1600/puttygen-3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="307" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5VJjHi56UBGhJXsS0z4GDvXtq8Z9nyBBGdI1CQ24yNN5MAKEw5b8yhOhbdk4EVxSL_o5WkLEd-wRNgUMTr7Sxb4pY9UceCEfGl1zAMdoHfQjvW7QZsVzC9JpGe55LaqebJdrPpTn9Bwc/s320/puttygen-3.png" width="320" /></a></div>
<br />
You can now configure a PuTTY session to use your private key, or <a href="http://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html">load it automatically on start up with Pageant</a>.<br />
<br />
<b>And done...</b><br />
<br />
You should now be able to log into your remote machine using public key authentication.<br />
<br />
Sometimes this won't work if your your home directory, your <code>.ssh</code>. directory, and other related files are group- or world-writable. If you're getting a "Permission denied (publickey)" error, try doing the following on your remote host:<br />
<br />
<pre>chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys</pre>
<br />
For more information check out these helpful links:<br />
<ul>
<li><a href="https://help.ubuntu.com/community/SSH/OpenSSH/Keys">Ubuntu Community Documentation > SSHOpenSSHKeys</a></li>
<li><a href="https://hkn.eecs.berkeley.edu/%7Edhsu/ssh_public_key_howto.html">SSH Public-Key Authentication HOWTO</a></li>
<li><a href="http://the.earth.li/%7Esgtatham/putty/0.61/htmldoc/Chapter8.html">PuTTY Docs > Using public keys for SSH authentication</a></li>
</ul>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-80247290192144146242011-11-09T16:27:00.000+07:002011-11-16T20:27:34.506+07:00Installing ESXi 5 on Intel DX58SO2 whitebox<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwjvZT8Pzr1TuZdDKrFKl1tUuCnD5K4Id3qxe5fdyT2gQx5P2VEPhxdy7uBvPO5E3r3tArcmHxDfke9pRr1HUwSEJucAbIw4lMpC4cti6i0OBOW_po9emaV7EuANYK9qxf4zHp5vk0D-M/s1600/vmware-server2-150.png" /></div>
Now that the <a href="http://blog.shvetsov.com/2011/11/my-new-server-mark-2.html">parts</a> have been assembled, <a href="http://blog.shvetsov.com/2011/11/make-vmware-esxi-5-liveusb-installer-on.html">vSphere 5 installer copied to USB</a> thumb drive and <a href="http://blog.shvetsov.com/2011/11/updating-bios-of-intel-dx58so2.html">BIOS updated</a>, it's time for installation.<br />
<br />
But first, a few BIOS tweaks. It appears that ESXi 5 installer refuses to run from my USB flash drive unless UEFI option in BIOS is enabled. To do that: <br />
<ul>
<li>go to your BIOS Setup (press F2 during boot)</li>
<li>go to "Boot" menu</li>
<li>change "UEFI boot" option to "Enabled"</li>
</ul>
While we're here, make sure USB boot is enabled as well. Then go to "Security" and enable all options related to Virtualization. Plug your USB drive, save and exit.<br />
<br />
By the way, there are two main reasons I went with this particular board. First, it's got two Gigabit network interfaces, meaning I can use it as firewall and/or router. Second, all components, including network and storage controller (except Intel's firmware RAID, a.k.a. Matrix Storage Technology), are supported out of the box. Once again big thanks to <a href="http://www.vm-help.com/">vm-help.com</a> for maintaining their <a href="http://www.vm-help.com/esx40i/esx40_whitebox_HCL.php">Whitebox HCL</a> and <a href="http://www.vm-help.com/forum/viewtopic.php?f=13&t=2822">forum</a>. With no drivers to worry about, installation just takes care of itself.<br />
<br />
While you're at the console, set your root password, configure your management interface and, a nice little addition to version 5, enable SSH and ESXi Shell.<br />
<br />
It's almost too easy, compared to <a href="http://blog.shvetsov.com/2010/04/installing-esxi-4-on-p6t-se-whitebox.html">my last install</a>.testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-67254513275572603772011-11-09T15:20:00.000+07:002011-11-09T16:23:10.126+07:00Updating BIOS of Intel DX58SO2Now that my <a href="http://blog.shvetsov.com/2011/11/my-new-server-mark-2.html">new server</a> is assembled I'd like to update BIOS to the latest version. Since I don't have Windows installed, I can't use the easy "Express BIOS Update" way. Luckily, Intel provides a way to do that without having OS installed at all. Thanks Intel, your commitment to making non-Windows user life easy will be rewarded with future purchases.<br />
<br />
Here are the instructions from DX58SO2 <a href="http://downloadmirror.intel.com/19451/eng/DX58SO2_ProductGuide01_English.pdf">manual</a>:<br />
<a name='more'></a><br />
<blockquote class="tr_bq">
<b>Updating the BIOS Using the F7 Function Key</b><br />
<br />
To use this BIOS update method:<br />
<ol>
<li>Download and save the Recovery BIOS (.BIO) file to a temporary directory.</li>
<li>Copy the .BIO to a USB thumb drive.</li>
<li>Plug the thumb drive into a USB port of the target computer.</li>
<li>Shut down the target computer.</li>
<li>Enable the F7 prompt display:</li>
<ol>
<li>Power the computer on.</li>
<li>Enter the BIOS Setup by pressing F2 during boot.</li>
<li>Go to the Advanced > Boot Configuration menu.</li>
<li>Enable Display F7 to Update BIOS</li>
<li>Press F10 to save and exit.</li>
</ol>
<li>During boot, when the F7 prompt is displayed, press F7 to enter the BIOS Flash Update tool.</li>
<li>Select the USB thumb drive and press Enter.</li>
<li>Select the .BIO file and press Enter</li>
<li>Confirm you want to update the BIOS by pressing Enter.</li>
<li>Wait 2-5 minutes for the update to complete.</li>
<li>Remove the thumb drive.</li>
<li>Restart the computer.</li>
</ol>
</blockquote>
The .BIO file could be found in <a href="http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=20372&lang=eng&DownloadType=BIOS">Intel's download center</a>. Either download the "Recovery BIOS Update" file or the "Iflash BIOS Update / Integrator Toolkit BIOS Files" bundle. The .BIO file inside the zip archive of the bundle is the same.testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.comtag:blogger.com,1999:blog-4330034378377463173.post-48223616566565193862011-11-08T15:02:00.000+07:002011-11-23T10:38:07.237+07:00Make VMware ESXi 5 LiveUSB installer on Mac OS X<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTcnUkZU0I7us0wcmH9rD0rm42VqWBGNiFHuauvOrZATF770yY_8_dqYjeF7mTPjPwcR76BiD0aAhs1w44tPjJx4H-4kC_G2tcsiMmuTrMNEjymZbIff1FFCsvPQwGTVn4cZRgi-V8FoA/s1600/unetbootin-150.png" /></div>
I have used <a href="http://unetbootin.sourceforge.net/">UNetbootin</a> in the past to <a href="http://blog.shvetsov.com/2010/06/quickly-create-vmware-esxi-4-liveusb.html">move VMware ESXi 4.0 installer to USB flash drive</a>, since then, however, this great tool now also works on Mac OS X (in addition to Linux and Windows). Since my MacBook Pro is the only <a href="http://blog.shvetsov.com/2011/11/my-new-server-mark-2.html">other computer</a> I own at the moment, this is a great new development. Now let's do the same thing with ESXi 5!<br />
<br />
One little note: UNetbootin won't see your USB drive if it's formatted as Mac OS Extended (Journaled) with GUID Partition Table. That's easy to fix with Disk Utility:<br />
<ol>
<li>select your USB drive from the list of devices on the left</li>
<li>click on the "Erase" tab on the right</li>
<li>choose "MS-DOS (FAT)" format</li>
<li>click "Erase" button</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGy2ynsm3VapI-gOL_LZszDG0FrI0Lbaa2iYULmif70-Xn8O4Pr5D0scLmoB8wSBxn5NKXffBqb7ASvmRPSpcRuC3X_2CYQuqngQfpv2ZHx3MjKfQCYQ3J9_yfi5tKQbGyRYYz8SPYTSs/s1600/erase-usb-with-diskutil.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="330" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGy2ynsm3VapI-gOL_LZszDG0FrI0Lbaa2iYULmif70-Xn8O4Pr5D0scLmoB8wSBxn5NKXffBqb7ASvmRPSpcRuC3X_2CYQuqngQfpv2ZHx3MjKfQCYQ3J9_yfi5tKQbGyRYYz8SPYTSs/s400/erase-usb-with-diskutil.jpg" width="400" /></a></div>
<br />
Now on to create the LiveUSB VMware vSphere 5 installer. Open UNetbotin, enter your Mac administrator password and then:<br />
<ol style="text-align: left;">
<li>select "Diskimage" option</li>
<li>browse to and select your previously downloaded VMware ESXi 5 Installer iso</li>
<li>make sure your installing onto correct USB drive</li>
<li>click "OK" and wait for wizard to complete</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOtdGMUoHYhV2HpBKo_lV92YMHp9Rqe41bbeGVcVUONdQLgNUgXzayV4Zapu59D2t4vOW2CNVOS6DuErE3U19HkTsDR3V91KzyxOxhZNhMJvJ6wlTWa7kRfLHobSLOUxvblGhcUxcMjKk/s1600/esxi-unetbootin.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOtdGMUoHYhV2HpBKo_lV92YMHp9Rqe41bbeGVcVUONdQLgNUgXzayV4Zapu59D2t4vOW2CNVOS6DuErE3U19HkTsDR3V91KzyxOxhZNhMJvJ6wlTWa7kRfLHobSLOUxvblGhcUxcMjKk/s400/esxi-unetbootin.jpg" width="400" /></a></div>
<br />
When wizard asks whether you want to overwrite <code>menu.c32</code> file, answer "No". This will allow you to boot directly into ESXi installer and bypass Syslinux default blue boot menu.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKicZm-5aIMTvqax_xpQ7iZzY1G0XVMBO5C25wuX155Nlid1Y2uFCqmrmwjrhRGdLD61_0NdUJ7_Mzq5-qE-geIzEraEVrSSe3_9aN0vruXTzhTIMcQUcvvkKzBW3KXdjZSP6h3CeCLfg/s1600/esxi-overwrite-no.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="291" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKicZm-5aIMTvqax_xpQ7iZzY1G0XVMBO5C25wuX155Nlid1Y2uFCqmrmwjrhRGdLD61_0NdUJ7_Mzq5-qE-geIzEraEVrSSe3_9aN0vruXTzhTIMcQUcvvkKzBW3KXdjZSP6h3CeCLfg/s400/esxi-overwrite-no.jpg" width="400" /></a></div>
<br />
Finish, eject and you're ready to install your sparkling new Hypervisor.<br />
<br />
<b>Update</b>: as you can see in comments below, it seems these steps are not quite sufficient to boot ESXi installer. I also had to <a href="http://blog.shvetsov.com/2011/11/installing-esxi-5-on-intel-dx58so2.html">enable UEFI boot option in BIOS</a>, while Raul <a href="http://avempace.posterous.com/installing-esxi-50-in-a-hp-proliant-turion-ii">added a kickstart configuration</a>.</div>testhttp://www.blogger.com/profile/05706376091708982536noreply@blogger.com