UPDATE
Puppetlabs now provides stdlib, which provides mechanisms for solving this and many other common problems. Check it out!
(The original post remains below, but using stdlib is a good idea!)
One of puppet's design goals is to be legible and useful to non-programmers. This is a laudable objective; not all sysadmins know how to write code, or are interested in doing so. However, this sometimes makes it necessary to... work around the limitations of the language.
Prime annoyance to myself: concatenating arrays for use in templates.
There's only one way in puppet language to concatenate arrays, and that's using the += operator on an array that was defined in a higher scope. Since puppet variables are immutable by design, this itself is actually a bit of chicanery: the += operator expands the variable from a higher scope, appends the data on the right side of the operator to that, and creates a new variable in the current scope with the same name as the one from the higher scope.
Sound confusing? Well, it is a bit. Here's an example of what it looks like:
$sshusers = [ "bob", "sally" ]
class ssh::accounts {
$sshusers += [ "tim", "thelma" ]
}
Within the scope of the ssh::accounts class, "sshusers" will be created as a new local variable, which expands to [ "bob", "sally", "tim", "thelma" ], and can no longer be modified. The original "sshusers" variable in the higher scope has not been modified.
So there you have it. That's how you append arrays in puppet. And that's the *only* way you can append to arrays in puppet.
At first, it might not be evident just how limiting this is. But consider the case that you have a lot of groups of users, defined in variables, and you want to use them all as elements in a single array:
$sysadmins = [ "bob", "sally" ]
$users = [ "tim", "thelma" ]
$dbas = [ "kip", "jim" ]
This is where things get nasty.
So, we know our += trick, but that can only combine *two* arrays at a time, so the only way to get all that junk into a single array is to chain += to get the mega-array we want. We now have a construct like this:
$sysadmins += $users
$dbas += $sysadmins
$sshusers = $dbas
Well, that works, but what if we want to get at only the DBAs for another template or definition? What if we want one definition that uses $sysadmins and $dbas, but another that uses $users and $sysadmins?
Basically, += does make this technically possible, but it makes it ugly. Wouldn't it be better if we could combine an arbitrary number of arrays in a single statement?
As far as I know, while there is no way to concatenate more than two arrays in puppet, you can still combine them into a single variable, like so:
$allusers = [ $dbas, $users, $sysadmins ]
All right, so this should trigger some warning bells. In a normal language, allusers would be an array which contains 3 other arrays. In puppet, though, there's not really a notion of nested arrays anywhere within the DSL itself; using $allusers as a variable in puppet definitions will work as if all the nested arrays have been expanded into a single array, which is what we really want.
For all practical purposes, within puppet's DSL, arrays that contain multiple sub-arrays function as if they are a single array containing all elements of the sub arrays.
Notice a very important qualification in my statement: "within puppet's DSL." This works fine when you're working within the puppet configuration itself, calling definitions, realizing users, etc; but if you try to use such a combined array in a template, it suddenly turns into a nested array again.
I find this duality very confusing; within the DSL, my variable is, in every way that I can interact with it, a single array of 6 strings. But if I try to use this array in a template, all of a sudden it's composed of 3 nested arrays which are in turn composed of 2 strings each.
Here's an example. Say you'd like to construct an sshd_config "AllowUsers" line in a template, which grants access to all of our users. AllowUsers should look like so:
AllowUsers bob sally tim thelma kip jim
Given that, you might define $sshusers like to combine your 3 arrays into a single variable:
$sshusers = [ $dbas, $users, $sysadmins ]
And call an ERB template that looks like:
<% sshusers.each { |i| -%>
<%= i + " "-%>
<% } ->
If sshusers is really an array of strings, this will do what you want: print out each element of the array followed by a space. But that's not what we get if we've combined our 3 smaller arrays, as we did above:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template ssh/sshd_config_new.erb: can't convert String into Array at /etc/puppet/modules/ssh/manifests/init.pp:79
That's not a very descriptive error (pointing out only the line in which the template is called), but it gives us a clue as to the type conversion problem at the root of our issue. What's happening here is that string concatenation fails since i is not a string. If you omit the '+ " "' stuff and run this template just printing 'i' as you iterate through the array, you see a list of all 6 items all bunched together. But the second you try to manipulate each element of the array, you realize that it's actually operating on 3 arrays, not 6 strings.
Bottom line is: you cannot combine more than two arrays in puppet to form an array of strings for use in a template, except by chaining += statements.
You can sort of work around this in some ways. One option would be to pass multiple variables to your template and use conditionals in the ERB to handle them properly. Thus, you have users1, users2, users3,... - but that leaves us with a rather unfortunate hack. Sure, it's fine for a small number of entries, but how many do you want to support? Wouldn't it be better if we could just get a darn array of strings?
I came up with a nasty hack: embedded in my ERB, I declared a function to "flatten" these nested arrays. It drilled down into the sub-arrays, iterated over them, and dumped each individual item into a single new array, which it returns. I then called that function whenever I needed the raw array.
UPDATE: a commenter posts out a much cleaner option than my original hack job, so I'm updating the post here in case anybody else should need it (there's really zero reason to use my old method, so it's stricken from the record):
AllowUsers <%= sshusers.flatten.join(' ') %>
Now, what do I get:
AllowUsers bob sally tim thelma kip jim
Success! Thanks Jay and duritong for pointing out this solution!
I still feel like this is a bit of a hack; why can't I properly concatenate arrays in the DSL? Why are not-concatenated arrays treated as if they were concatenated in the DSL? The workaround here gets the job done, but to me it was not obvious at all what was going on; this behavior should at least be referred to in the documentation.
Comments
Jay @ Fri Jun 18 17:12:02 -0400 2010
AllowUsers <%= sshusers.flatten.join(' ') %>
^this would work too. Ruby's got your back, even if Puppet doesn't.
duritong @ Mon May 10 16:10:24 -0400 2010
<% sshusers.flatten.each { |i| -%>
<%= i + " "-%>
<% } ->
Jeremy @ Wed Aug 11 15:57:06 -0400 2010
Ah, the flatten.join stuff certainly looks much cleaner in a template! I will update the main post.
Simon J Mudd @ Mon Apr 16 07:04:57 -0400 2012
I came across your post today and am frustrated by the same problem.
However, the solution you propose of fixing this in the template doesn't always apply if you don't have direct access to the template in the class you are modifying.
I am modifying some classes which configure /etc/my.cnf for MySQL, the actual template creation is done in a class I call where I provide some special parameters that need setting up.
Some of these parameters are provided to the class I am working in as an array. Until now that array has been passed down to the "templator" class (it does more than just that) and that worked fine. Now depending on some other conditions I need to modify this array and due to the reasons stated in your blog post I really can't.
In a language like perl solving this is trivial:
@sshusers = ( @dbas, @users, @sysadmins ) # in perl syntax but equally you can do
@sshusers = ( @dbas, 'simon', @sysadmins ) # and not have a problem if 'simon' is a string.
I guess that adding some sort of flatten() function where you can provide an arbitrary list of arrays or scalars would solve this but that needs to be done in the DSL where it's actually needed and where you don't have fuller access to the ruby language which you do when changing templates.
Still working on a workable solution without having to rearrange the puppet recipes excessively.
Jeremy @ Fri Apr 27 10:51:19 -0400 2012
Simon:
One consideration is that you may assign templates to variables within the DSL. This means you can do something like:
# create a flattening template that generates the string you want; let's say this template returns "foo|bar|baz"
$foo = template("my_module/flatten.erb")
# now foo is a string, not an array, but you can use the DSL's 'split' function to make it an array
$fooarray = split($foo, '|')
Annoyance here being you would need a template solely to flatten each array you need to do this with. There *may* be possibilities in this respect using the new Ruby DSL in Puppet 2.6+
Note however! In puppet 2.8 building lists of variables is going to become awkward, since variables will no longer fall through to lower scopes. So you might need to do some extensive refactoring of any code that conditionally builds or modifies variables.
Benjamin Abbott-Scott @ Tue Jul 31 19:07:56 -0400 2012
And how about when you're not using a template? Like, say, managing logins. If you do:
$sshusers = [ $dbas, $users, $sysadmins ],
...
user { [$sshusers] :
...
}
and someone added 'sally' to both $users and $sysadmins, suddenly everything fails because user resource 'sally' is now defined twice. Horrid death on a cracker.
mc0e @ Tue Sep 04 11:24:20 -0400 2012
Take a look at the functions provided by puppetlabs-stdlib.
In simple cases you can use the flatten function directly in your manifest:
$sysadmins = [ "bob", "sally" ]
$users = [ "tim", "thelma" ]
$dbas = [ "kip", "jim" ]
$sshusers = flatten([$sysadmins,$users,$dbas])
Benjamin Abbott-Scott points out the problem of duplicates. There's a 'unique' function that's useful for that:
$sshusers = unique(flatten([$sysadmins,$users,$dbas]))
Be aware though that flatten will flatten arrays nested many layers deep, so it's not the same thing as concatenating arrays. If you need to that in puppet though, there's nothing to stop you writing a custom function to do it. It's not all that hard.
Jeremy @ Mon Feb 04 13:29:44 -0500 2013
I've updated the blog entry with a link to stdlib. That's definitely the easiest way to solve this particular issue these days!