Jekyll2018-07-26T10:10:23+00:00http://beez.ly/WebsiteMy Blog Posts - Andrew BeresfordAuthenticating Azure users with Hashicorp Vault2018-07-25T00:00:00+00:002018-07-25T00:00:00+00:00http://beez.ly/2018/07/25/authenticating-azure-users-with-hashicorp-vault<p>Hashicorp <a href="https://groups.google.com/forum/#!topic/hashicorp-announce/icgKzpswe-s">released Vault 0.10.4</a> yesterday and one of the new features is a <a href="https://www.vaultproject.io/docs/auth/jwt.html">plugin to support JWT/OIDC</a> as an auth method. Azure has an OIDC endpoint that you can use to authenticate against and if you are logged in with the Azure CLI tools you can easily get an access token with <code class="highlighter-rouge">az account get-access-token</code>. Here’s how to get it working.</p>
<!--more-->
<p>First thing is to ensure you are running vault 0.10.4. Then enable the jwt auth method.</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>vault auth <span class="nb">enable </span>jwt
Success! Enabled jwt auth method at: jwt/</code></pre></figure>
<p>Then configure the JWT endpoint. Be sure to replace <code class="highlighter-rouge">{tenant_id}</code> with your Azure tenant ID.</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>vault write auth/jwt/config <span class="nv">oidc_discovery_url</span><span class="o">=</span><span class="s1">'https://sts.windows.net/{tenant_id}/'</span>
Success! Data written to: auth/jwt/config</code></pre></figure>
<p>Configure a role to log in as. The “audience” is dictated by what is provided by the Azure CLI. The “upn” claim is used by Vault to calculate an identity and Microsoft claim <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-token-and-claims">it stores the User Principal Name</a> of the authenticating user.</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>vault write auth/jwt/role/test-role <span class="nv">policies</span><span class="o">=</span><span class="s1">'dev'</span> <span class="nv">bound_audiences</span><span class="o">=</span><span class="s1">'https://management.core.windows.net/'</span> <span class="nv">user_claim</span><span class="o">=</span><span class="s1">'upn'</span>
Success! Data written to: auth/jwt/role/test-role</code></pre></figure>
<p>And that’s it! In this example below, I get a vault token using the role created above and extract the token from the Azure CLI’s output using <a href="https://stedolan.github.io/jq/">jq</a>.</p>
<figure class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$ </span>vault write auth/jwt/login <span class="nv">role</span><span class="o">=</span>test-role <span class="nv">jwt</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>az account get-access-token <span class="nt">-o</span> json | jq <span class="nt">-r</span> .accessToken<span class="k">)</span><span class="s2">"</span>
Key Value
<span class="nt">---</span> <span class="nt">-----</span>
token 4fa15547-ae06-ea74-5189-7eff73a06a0c
token_accessor b7a01c99-3067-cd01-0bc7-cbee75b2aab5
token_duration 768h
token_renewable <span class="nb">true
</span>token_policies <span class="o">[</span><span class="s2">"default"</span> <span class="s2">"dev"</span><span class="o">]</span>
identity_policies <span class="o">[]</span>
policies <span class="o">[</span><span class="s2">"default"</span> <span class="s2">"dev"</span><span class="o">]</span>
token_meta_role test-role</code></pre></figure>Hashicorp released Vault 0.10.4 yesterday and one of the new features is a plugin to support JWT/OIDC as an auth method. Azure has an OIDC endpoint that you can use to authenticate against and if you are logged in with the Azure CLI tools you can easily get an access token with az account get-access-token. Here’s how to get it working.Provisioning VMs Using Ephemeral SSH Keys in Terraform2018-04-11T00:00:00+00:002018-04-11T00:00:00+00:00http://beez.ly/2018/04/11/provisioning-virtual-machines-using-ephemeral-ssh-keys-in-terraform<p>When provisioning Linux VMs in the cloud, it is useful to use SSH keys to bootstrap the VMs with some initial configuration. This post explains how to provision your VMs using a disposable SSH key.</p>
<!--more-->
<p>Terraform helps by providing the <code class="highlighter-rouge">tls_private_key</code> resource type.</p>
<figure class="highlight"><pre><code class="language-hcl" data-lang="hcl">resource "tls_private_key" "bootstrap_private_key" {
algorithm = "RSA"
rsa_bits = "4096"
}</code></pre></figure>
<p>In the terraform VM resource I configure the VM to use the public key. In this example, I’m using Azure so the resource would be an <code class="highlighter-rouge">azurerm_virtual_machine</code>. The value of <code class="highlighter-rouge">path</code> in the ssh_keys attribute must be set to the path of the admin user’s <code class="highlighter-rouge">authorized_hosts</code> file. The <code class="highlighter-rouge">key_data</code> is set to the <code class="highlighter-rouge">public_key_openssh</code> attribute of the <code class="highlighter-rouge">bootstrap_private_key</code> resource created above.</p>
<figure class="highlight"><pre><code class="language-hcl" data-lang="hcl">resource "azurerm_virtual_machine" "demo_vm" {
<attributes not relevant to this example>
os_profile {
admin_username = "setup"
}
os_profile_linux_config {
disable_password_authentication = true
ssh_keys = [{
path = "/home/setup/.ssh/authorized_keys"
key_data = "${chomp(tls_private_key.bootstrap_private_key.public_key_openssh)}"
}]
}
}</code></pre></figure>
<p>Any provisioners that need to connect to the remote host can then be configured to use the private key from the <code class="highlighter-rouge">tls_private_key</code> resource.</p>
<figure class="highlight"><pre><code class="language-hcl" data-lang="hcl"> connection {
type = "ssh"
host = "${azurerm_public_ip.bastion_public_ip.ip_address}"
user = "${var.os_admin_username}"
private_key = "${tls_private_key.bootstrap_private_key.private_key_pem}"
}</code></pre></figure>
<p>In this case terraform copies across some bootstrap code onto the VM which provisions my user accounts. Finally, it deletes the admin user’s <code class="highlighter-rouge">authorized_keys</code> file to ensure that the ephemeral key cannot be used to log in again.</p>
<figure class="highlight"><pre><code class="language-hcl" data-lang="hcl"> provisioner "file" {
source = "${path.module}/some-files"
destination = "/tmp/some-files"
}
provisioner "remote-exec" {
inline = [
"sudo bash /tmp/some-files/bootstrap.sh",
"sudo rm -f /home/setup/.ssh/authorized_keys",
]
}</code></pre></figure>
<h1 id="limitations">Limitations</h1>
<p>Terraform will store the private key in the state file, and in the event that a VM resource is destroyed and re-created, the same key will be used to provision the resource. Of course, the state file should be stored securely anyway, but in an ideal world it would be possible to only define this resource for the duration of the <code class="highlighter-rouge">apply</code>. Maybe some new features in terraform in the future will help with this.</p>
<p>It is possible to partially workaround this, by always tainting the <code class="highlighter-rouge">tls_private_key</code> after an apply, which would guarantee that the key is recreated on the next <code class="highlighter-rouge">apply</code> execution.</p>When provisioning Linux VMs in the cloud, it is useful to use SSH keys to bootstrap the VMs with some initial configuration. This post explains how to provision your VMs using a disposable SSH key.Migrating an LVM logical volume onto a new block device without downtime2014-06-02T00:00:00+00:002014-06-02T00:00:00+00:00http://beez.ly/2014/06/02/migrating-an-lvm-logical-volume-onto-a-new-block-device-without-downtime<p>Recently we had a problem at work that required us to move a filesystem from local disk onto an iSCSI device. Normally you could just create a new filesystem on the iSCSI device, use a tool like rsync to move the data across and remount the new filesystem in place of the old one.</p>
<!--more-->
<p>Unfortunately, this filesystem is in use nearly all the time and it is difficult to schedule downtime on this system, so we needed to come up with a different strategy.</p>
<p>Luckily, the filesystem is built on disks which use Logical Volume Manager, so we were able to use that to migrate across to the new iSCSI storage.</p>
<h2 id="step-1---creating-the-new-physical-volume">Step 1 - Creating the new Physical Volume</h2>
<p>This should be familiar to those using LVM. Add the new partition as a physical volume using the <code>pvcreate</code> command;</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pvcreate <newPV>
</code></pre></div></div>
<p>In this case, the disk I’m adding is <code>sdb1</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# pvcreate /dev/sdb1
Writing physical volume data to disk "/dev/sdb1"
Physical volume "/dev/sdb1" successfully created
</code></pre></div></div>
<h2 id="step-2---add-the-new-physical-volume-to-the-existing-volume-group">Step 2 - Add the new Physical Volume to the existing Volume Group</h2>
<p>On this machine, we have a volume group called <code>VolGroup00</code>. You can see information about the existing VGs using <code>vgdisplay</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# vgdisplay
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 14
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 6
Open LV 5
Max PV 0
Cur PV 2
Act PV 2
VG Size 29.88 GB
PE Size 32.00 MB
Total PE 956
Alloc PE / Size 699 / 21.84 GB
Free PE / Size 257 / 8.03 GB
VG UUID uTkXkf-MNXf-dYoQ-DRDS-s5y1-73kX-H0ylTH
</code></pre></div></div>
<p>I add the new PV to this VG using <code>vgextend</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vgextend <VG> <newPV>
</code></pre></div></div>
<p>This produces:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# vgextend VolGroup00 /dev/sdb1
Volume group "VolGroup00" successfully extended
</code></pre></div></div>
<h2 id="step-3---mirror-the-existing-logical-volume-onto-the-new-physical-volume">Step 3 - Mirror the existing logical volume onto the new physical volume</h2>
<p>Using <code>lvconvert</code> we can change the properties of the existing logical volume to require mirroring. We can use this to move data across to the new device without stopping the filesystem and unmounting it.</p>
<p>First we can take a look at the information for the LV we’re migrating using <code>lvdisplay</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvdisplay <LV>
</code></pre></div></div>
<p>In this case I’m interested in the PV at /dev/Volgroup00/webusers</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# lvdisplay /dev/VolGroup00/webusers
--- Logical volume ---
LV Name /dev/VolGroup00/webusers
VG Name VolGroup00
LV UUID cOAHtW-YWND-hItx-eYbI-WZAb-2CID-Oe2qZM
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 64
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
</code></pre></div></div>
<p>Using <code>lvconvert</code>, we tell LVM to make this a mirrored volume using the new PV we added a few moments ago:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvconvert -m1 --mirrorlog core <VG>/<LV> <newPV>
</code></pre></div></div>
<p>This tells LVM to create 1 mirror copy (-m1), using an in-memory log. In-memory is fine for our purposes as we’re only using this mirror to store data temporarily whilst it is migrated.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# lvconvert -m1 --mirrorlog core VolGroup00/webusers /dev/sdb1
VolGroup00/webusers: Converted: 7.8%
VolGroup00/webusers: Converted: 54.7%
VolGroup00/webusers: Converted: 100.0%
</code></pre></div></div>
<p>If we now examine the LV with <code>lvdisplay</code> we’ll see a significant change:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# lvdisplay /dev/VolGroup00/webusers
--- Logical volume ---
LV Name /dev/VolGroup00/webusers
VG Name VolGroup00
LV UUID cOAHtW-YWND-hItx-eYbI-WZAb-2CID-Oe2qZM
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 64
Mirrored volumes 2
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
</code></pre></div></div>
<p>The LV now has two mirrored volumes and the data is now on the iSCSI device.</p>
<h2 id="step-4---detaching-the-original-physical-volume">Step 4 - Detaching the original physical volume</h2>
<p>We can use use <code>lvconvert</code> again to remove the old physical volume by removing the mirror property from the LV.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvconvert -m0 <VG>/<LV> <oldPV>...
</code></pre></div></div>
<p>In my case I am removing /dev/sda2 and /dev/sda3. If you are migrating from multiple old PVs, you can specify them all.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# lvconvert -m0 VolGroup00/webusers /dev/sda2 /dev/sda3
Logical volume webusers converted.
</code></pre></div></div>
<p><code>lvdisplay -m</code> should now show that the data is on the new PV.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@voltaire [~]# lvdisplay -m /dev/VolGroup00/webusers
--- Logical volume ---
LV Name /dev/VolGroup00/webusers
VG Name VolGroup00
LV UUID cOAHtW-YWND-hItx-eYbI-WZAb-2CID-Oe2qZM
LV Write Access read/write
LV Status available
# open 0
LV Size 2.00 GB
Current LE 64
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
--- Segments ---
Logical extent 0 to 63:
Type linear
Physical volume /dev/sdb1
Physical extents 0 to 63
</code></pre></div></div>
<h2 id="step-5---remove-the-old-pv-from-the-vg-and-destroy-it">Step 5 - Remove the old PV from the VG and destroy it</h2>
<p>Once you are no longer using the old PV, you can remove it from the volume group and destroy it.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vgreduce <VG> <oldPV>
pvremove <oldPV>
</code></pre></div></div>
<p>Finished!</p>Recently we had a problem at work that required us to move a filesystem from local disk onto an iSCSI device. Normally you could just create a new filesystem on the iSCSI device, use a tool like rsync to move the data across and remount the new filesystem in place of the old one.Building a development CAS environment with CASinoApp and casino-test_authenticator2014-05-15T00:00:00+00:002014-05-15T00:00:00+00:00http://beez.ly/2014/05/15/building-a-development-cas-environment-with-casinoapp-and-casino-test_authenticator<p>Sometimes it’s useful to be able to run a development CAS instance which allows you to
authenticate test users easily without createing lots of test accounts. I wrote
<a href="http://github.com/beezly/casino-test-authenticator">casino-test_authenticator</a> for exactly this purpose and you can get it running 2-3 minutes with <a href="http://casino.rbcas.com/">Casino</a></p>
<!--more-->
<p>First get a copy of CASinoApp</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/rbCAS/CASinoApp.git
cd CASinoApp
</code></pre></div></div>
<p>Add the following to CasinoApp’s Gemfile</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem 'casino-test_authenticator'
</code></pre></div></div>
<p>Then configure config/cas.yml to include the test authenticator in the development environment</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">development</span><span class="pi">:</span>
<span class="na">frontend</span><span class="pi">:</span>
<span class="na">sso_name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">CASino'</span>
<span class="na">footer_text</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Powered</span><span class="nv"> </span><span class="s">by</span><span class="nv"> </span><span class="s"><a</span><span class="nv"> </span><span class="s">href="http://rbcas.com/">CASino</a>'</span>
<span class="na">authenticators</span><span class="pi">:</span>
<span class="na">test_authenticator</span><span class="pi">:</span>
<span class="na">authenticator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Test"</span>
<span class="na">options</span><span class="pi">:</span></code></pre></figure>
<p>And configure the database in config/database.yml</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">development</span><span class="pi">:</span>
<span class="na">adapter</span><span class="pi">:</span> <span class="s">sqlite3</span>
<span class="na">database</span><span class="pi">:</span> <span class="s">db/development.sqlite3</span>
<span class="na">pool</span><span class="pi">:</span> <span class="s">5</span>
<span class="na">timeout</span><span class="pi">:</span> <span class="s">5000</span></code></pre></figure>
<p>Get all the prerequisites installed</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle
</code></pre></div></div>
<p>Load the database</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle exec rake db:schema:load
</code></pre></div></div>
<p>And run the application</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails s
</code></pre></div></div>
<p>Or use -p <port number=""> to run it on a different port (the default is 3000)</port></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails s -p 9999
</code></pre></div></div>
<p>You should be able to access http://localhost:3000/ and log in when username==password</p>Sometimes it’s useful to be able to run a development CAS instance which allows you to authenticate test users easily without createing lots of test accounts. I wrote casino-test_authenticator for exactly this purpose and you can get it running 2-3 minutes with Casino