Search Posts:



March 2010

1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31


Puppetlabs now provides stdlib, which provides mechanisms for solving this and many other common problems. Check it out!

(The original post remains below, but using stdlib is a good idea!)

One of puppet's design goals is to be legible and useful to non-programmers. This is a laudable objective; not all sysadmins know how to write code, or are interested in doing so. However, this sometimes makes it necessary to... work around the limitations of the language.

Prime annoyance to myself: concatenating arrays for use in templates.

There's only one way in puppet language to concatenate arrays, and that's using the += operator on an array that was defined in a higher scope. Since puppet variables are immutable by design, this itself is actually a bit of chicanery: the += operator expands the variable from a higher scope, appends the data on the right side of the operator to that, and creates a new variable in the current scope with the same name as the one from the higher scope.

Sound confusing? Well, it is a bit. Here's an example of what it looks like:

$sshusers = [ "bob", "sally" ]
class ssh::accounts {
$sshusers += [ "tim", "thelma" ]

Within the scope of the ssh::accounts class, "sshusers" will be created as a new local variable, which expands to [ "bob", "sally", "tim", "thelma" ], and can no longer be modified. The original "sshusers" variable in the higher scope has not been modified.

So there you have it. That's how you append arrays in puppet. And that's the *only* way you can append to arrays in puppet.

At first, it might not be evident just how limiting this is. But consider the case that you have a lot of groups of users, defined in variables, and you want to use them all as elements in a single array:

$sysadmins = [ "bob", "sally" ]
$users = [ "tim", "thelma" ]
$dbas = [ "kip", "jim" ]

This is where things get nasty.

So, we know our += trick, but that can only combine *two* arrays at a time, so the only way to get all that junk into a single array is to chain += to get the mega-array we want. We now have a construct like this:

$sysadmins += $users
$dbas += $sysadmins
$sshusers = $dbas

Well, that works, but what if we want to get at only the DBAs for another template or definition? What if we want one definition that uses $sysadmins and $dbas, but another that uses $users and $sysadmins?

Basically, += does make this technically possible, but it makes it ugly. Wouldn't it be better if we could combine an arbitrary number of arrays in a single statement?

As far as I know, while there is no way to concatenate more than two arrays in puppet, you can still combine them into a single variable, like so:

$allusers = [ $dbas, $users, $sysadmins ]

All right, so this should trigger some warning bells. In a normal language, allusers would be an array which contains 3 other arrays. In puppet, though, there's not really a notion of nested arrays anywhere within the DSL itself; using $allusers as a variable in puppet definitions will work as if all the nested arrays have been expanded into a single array, which is what we really want.

For all practical purposes, within puppet's DSL, arrays that contain multiple sub-arrays function as if they are a single array containing all elements of the sub arrays.

Notice a very important qualification in my statement: "within puppet's DSL." This works fine when you're working within the puppet configuration itself, calling definitions, realizing users, etc; but if you try to use such a combined array in a template, it suddenly turns into a nested array again.

I find this duality very confusing; within the DSL, my variable is, in every way that I can interact with it, a single array of 6 strings. But if I try to use this array in a template, all of a sudden it's composed of 3 nested arrays which are in turn composed of 2 strings each.

Here's an example. Say you'd like to construct an sshd_config "AllowUsers" line in a template, which grants access to all of our users. AllowUsers should look like so:

AllowUsers bob sally tim thelma kip jim

Given that, you might define $sshusers like to combine your 3 arrays into a single variable:

$sshusers = [ $dbas, $users, $sysadmins ]

And call an ERB template that looks like:

<% sshusers.each { |i| -%>
<%= i + " "-%>
<% } ->

If sshusers is really an array of strings, this will do what you want: print out each element of the array followed by a space. But that's not what we get if we've combined our 3 smaller arrays, as we did above:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template ssh/sshd_config_new.erb: can't convert String into Array at /etc/puppet/modules/ssh/manifests/init.pp:79

That's not a very descriptive error (pointing out only the line in which the template is called), but it gives us a clue as to the type conversion problem at the root of our issue. What's happening here is that string concatenation fails since i is not a string. If you omit the '+ " "' stuff and run this template just printing 'i' as you iterate through the array, you see a list of all 6 items all bunched together. But the second you try to manipulate each element of the array, you realize that it's actually operating on 3 arrays, not 6 strings.

Bottom line is: you cannot combine more than two arrays in puppet to form an array of strings for use in a template, except by chaining += statements.

You can sort of work around this in some ways. One option would be to pass multiple variables to your template and use conditionals in the ERB to handle them properly. Thus, you have users1, users2, users3,... - but that leaves us with a rather unfortunate hack. Sure, it's fine for a small number of entries, but how many do you want to support? Wouldn't it be better if we could just get a darn array of strings?

I came up with a nasty hack: embedded in my ERB, I declared a function to "flatten" these nested arrays. It drilled down into the sub-arrays, iterated over them, and dumped each individual item into a single new array, which it returns. I then called that function whenever I needed the raw array.

UPDATE: a commenter posts out a much cleaner option than my original hack job, so I'm updating the post here in case anybody else should need it (there's really zero reason to use my old method, so it's stricken from the record):

AllowUsers <%= sshusers.flatten.join(' ') %>

Now, what do I get:

AllowUsers bob sally tim thelma kip jim

Success! Thanks Jay and duritong for pointing out this solution!

I still feel like this is a bit of a hack; why can't I properly concatenate arrays in the DSL? Why are not-concatenated arrays treated as if they were concatenated in the DSL? The workaround here gets the job done, but to me it was not obvious at all what was going on; this behavior should at least be referred to in the documentation.

Here's the moon, a waxing gibbous from Saturday night; read on for details.

From Nature

My gear: Sony A700, Minolta 500mm f/8 Reflex (a fixed aperture catadioptric lens), tripod.

I found getting good shots more difficult than I had expected. I'm relatively new to photography and while I understand the basics, trying to shoot the moon pretty much causes all those automatic bells and whistles on your camera to become useless.

For starters, the metering system isn't very useful; if you leave it on matrix or center weighted with a lens of this length, it's going to blow out highlights badly due to all the black in the frame. Spot metering is closer to right, but it's still sketchy. The best technique I found so far is going full on manual exposure.

I found that the best results were with shutter speeds in the 1/125 range at ISO 200 (at least, this was the best when the moon was about halfway between the horizon and directly above - it should put off more light the higher it is in the sky). Incidentally, this isn't far off from the "sunny 16" rule, which makes perfect sense when you think about it; the moon is not a source of light in and of itself, rather it's reflected sunlight, so it's logical to use the calculation based on a sunny day. Sunny 16 underexposes by about 2-3 stops in my tests, due to the impact of atmosphere.

Now 1/125 second is going to be difficult to handhold with a 500mm lens. When hand holding, you need the ISO jacked up to around 1600 or better to get shutter speeds high. I try to avoid going that high if I can, so I used a tripod and longer exposure.

Automatic white balance is equally sketchy. It actually did OK sometimes, but it was hit or miss. You either need to set the WB manually, or just plan on fixing it in post processing (I chose the latter).

Now, depending on how accurate your exposure is, you have some work to do in software. The JPEG engine on my A700 did a really poor job with contrast, so I used RAW. I use ufraw and the GIMP; at 1/125 second all I really needed to do was bring up the black point to enhance contrast on the moon's surface. If you underexpose (as I did in this sample) you have to bring the white point down as well.

I had to use the GIMP and ufraw for this since Picasa's contrast adjustments were inadequate. "Auto contrast" is a disaster, but worse is that Picasa "guesses" some initial EV values when using RAW, and those guesses were already clipping highlights. It's not even possible to bring these back down to proper levels within Picasa!

I also applied some unsharp mask (.4 as the value) in the GIMP. I think I'm hitting the limitations of the lens in terms of resolving power, and it just can't fill the A700's entire 12MP sensor. This is another good reason to try and avoid high ISO, as USM will sharpen noise if it exists.

So anyway... that's shooting the moon! It's not hard when you know how to do it, but it took me a bit of time to learn.