Getting Bashed by Dynamic Arrays

Written by Ludvik Jerabek on August 24, 2015 in Linux, Technology, Tutorials

The reason most people love Linux is because of the ability to quickly prototype a system; due to the plethora of free software, command line tools, tool kits, and scripting languages available to choose from. Recently I was working on a script written in the Bourne Again SHell commonly known as (bash). The primary reason for choosing Bash was because it’s widely available and it supports arrays and associative arrays; unlike it’s predecessor the Bourne Shell aka (bs) … just kidding it’s actually (sh).

The script would use curl to make web-services calls, xmllint for XML parsing, and some other commonly used Linux tools such as grep. This command line tool would essentially pull XML based software catalogs which contain software versions, upgrade paths, and other content which could be downloaded via the web service. Parsing data via pipes to xmllint, sed, awk, and grep is an easy task, but storing that data into data purely Bash data structures/types presents some interesting challenges. The goal in the end is to dynamically build a software catalog which allows the user to interact with Text User Interface (TUI) to download the needed components for installation later. Everything from the software classes, version names, and upgrade paths is unknown prior to the initial web-service call.

Any well versed programmer by now has thought to themselves, “It’s easy you have arrays and associative arrays how hard could that be?”

Let’s cover the basics around arrays without introducing anything dynamic just dealing with spaces.

# Define the array
declare -a array

# The following line is good it will result in a single element being added array[0]="good"
array+=(goood)

# The following line will result in array[0]=aa,&nbsp;array[1]=bb,&nbsp;array[0]=cc
array+=(aa bb cc)

# The following line will result in array[0]="aa bb cc"
array+=("aa bb cc")

# Set a value at at index 0
array[0]=good

# Set a value at a &nbsp;index 0 with a space
array[0]="good work"
Code language: PHP (php)

Let’s cover the basics around associative arrays without introducing anything dynamic just dealing with spaces.

#Define the associative array
declare -A aarray

# The following line is good it will result in a single element where the key is "key"  and the value is "good"
aarray[key]=good

# The following line will fail because you cannot assign list to array member
aarray[key]=(aa bb cc)

# The following will work because it assigns the entire string to  aarray[key]
aarray[key]="(aa bb cc)"

# If you execute the following command
echo ${aarray[key]}

# The result will be the text stored
(aa bb cc)Code language: PHP (php)

This means that neither arrays or associative arrays support list assignment at an index or key respectively thus ruling out the following options:

# These wont work!!!
${array[0][0]}
${aarray[key][0]}
Code language: PHP (php)

For the most part everything else works as one would expect, but there is no native support for multi-dimensional arrays although there are plenty of ways to simulate this behavior all of which can get quite dirty before even adding dynamic variables. When creating a dialog driven system it is going to be necessary be able to directly map an option index to an array index as shown below:

Microsoft
IBM
Solar
Exit

Please select an option: 1

These options would naturally lead to to other dynamically generated options

Microsoft Software

Microsoft Office
Microsoft Visio
Micorsoft Visual Studio
Back to the previous menu

Please select an option: 1

Microsoft Software

Office 95
Office 98
Office 2005
Office 2011
Back to the previous menu

These options would once again lead to other dynamically populated data which would result in an acquisition. The same is true when using ‘dialog’ to for presentation of menus, checked lists, or radio lists.

dialog –ascii-lines –menu “Choose software vendor:” 20 80 5 1 Microsoft 2 IBM 3 “Sun Microsystems”

At this point in time it’s probably worth showing some of the options for dynamic variables using ‘declare’

# Simulated array with dynamic variables
index=0
declare value_$index="value1"
index=$((index+1))
declare value_$index="value2"
index=$((index+1))

# The above lines create two variables
value_0=value1
value_1=value2

# However when you echo 'echo \$value_$((index))' will not work and results in the following output
$value_1

# To extract the actual value we need to use 'eval' which results in "value2" being displayed
eval echo \$value_$((index-1))

# It would be possible to loop through the values with the following for loop
for (( i=0 ; i < $index ; i++ )) 
do 
    eval echo \$value_$i
doneCode language: PHP (php)

Now that we have seen how to implement this simulated dynamic data type with ‘declare’ we can try to do it without

# Simulated array with dynamic variables without declare
# The following will NOT work producing the following error: value_0=value1: command not found
index=0
value_$index="value1"

# Once again we need to use eval to solve this problem
index=0
eval value_$index="value1"
index=$((index+1))
eval value_$index="value2"
index=$((index+1))

# To extract the actual value we need to use 'eval' which results in "value2" being displayed
eval echo \$value_$((index-1))

# It would be possible to loop through the values with the following for loop
for (( i=0 ; i < $index ; i++ )) 
do 
    eval echo \$value_$i
done
Code language: PHP (php)

The next step is to apply what we have learned about dynamic variable creation and apply it to dynamic variable array creation using declare

# Dynamically named array
my_variable_name="dyn_arr_names"
declare -a $my_variable_name

# Adding by index to the array eg. dyn_arr_names[0]="bob"
eval $my_variable_name[0]="bob"

# Adding by pushing onto the array eg. dyn_arr_names+=(robert)
eval $my_variable_name+=\(robert\)

# Print value stored at index indirect
echo ${!my_variable_name[0]}

# Print value stored at index
eval echo \${$my_variable_name[0]}

# Get item count
eval echo \${#my_variable_name[@]}Code language: PHP (php)

Now that we have seen how to implement this dynamic variable array with ‘declare’ we can try to do it without

# Dynamically named array
my_variable_name="dyn_arr_names"
eval $my_variable_name=\(\)

# Adding by index to the array eg. dyn_arr_names[0]="bob"
eval $my_variable_name[0]="bob"

# Adding by pushing onto the array eg. dyn_arr_names+=(robert)
eval $my_variable_name+=\(robert\)

# Print value stored at index indirect
echo ${!my_variable_name[0]}

# Print value stored at index
eval echo \${$my_variable_name[0]}

# Get item count
eval echo \${#$my_variable_name[@]}Code language: PHP (php)

It pretty easy to see that this can get quite complicated once you need to dynamically pass the variable name, index, and value. Especially when you start to consider the limitations of indirect array references using ‘!’ it is not possible to get the number of items stored in the array. It’s best to encapsulate that in some functions to make life a bit easier. It also allows us to make sure that variables adhere to the proper naming convention. The functions below allow creation, insertion, retrieval, and assignment very easy. It should be noted that you can use ‘declare’ within a function but you will need to pass -ag to make the variable scope global.

# Dynamically create an array by name
function arr() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
     # The following line can be replaced with 'declare -ag $1=\(\)'
     # Note: For some reason when using 'declare -ag $1' without the parentheses will make 'declare -p' fail
	eval $1=\(\)
}

# Insert incrementing by incrementing index eg. array+=(data)
function arr_insert() {	
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	eval $1[\$\(\(\${#${1}[@]}\)\)]=\$2
}

# Update an index by position
function arr_set() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	eval ${1}[${2}]=\${3}
}

# Get the array content ${array[@]}
function arr_get() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	eval echo \${${1}[@]}
}

# Get the value stored at a specific index eg. ${array[0]}  
function arr_at() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	[[ ! "$2" =~ ^(0|[-]?[1-9]+[0-9]*)$ ]] && { echo "Array index must be a number" 1>&2 ; return 1 ; }
	local v=$1
	local i=$2
	local max=$(eval echo \${\#${1}[@]})
	# Array has items and index is in range
	if [[ $max -gt 0 && $i -ge 0 && $i -lt $max ]]
	then 
		eval echo \${$v[$i]}
	fi
}

# Get the value stored at a specific index eg. ${array[0]}  
function arr_count() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable " 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	local v=${1}
	eval echo \${\#${1}[@]}
}



array_names=(bob jane dick)

for name in "${array_names[@]}"
do
	arr dyn_$name
done

echo "Arrays Created"
declare -a | grep "a dyn_"

# Insert three items per array
for name in "${array_names[@]}"
do
	echo "Inserting dyn_$name abc"
	arr_insert dyn_$name "abc"
	echo "Inserting dyn_$name def"
	arr_insert dyn_$name "def"
	echo "Inserting dyn_$name ghi"
	arr_insert dyn_$name "ghi"
done

for name in "${array_names[@]}"
do
	echo "Setting dyn_$name[0]=first"
	arr_set dyn_$name 0 "first"
	echo "Setting dyn_$name[2]=third"
	arr_set dyn_$name 2 "third"
done 

declare -a | grep "a dyn_"

for name in "${array_names[@]}"
do
	arr_get dyn_$name
done


for name in "${array_names[@]}"
do
	echo "Dumping dyn_$name by index"
	# Print by index
	for (( i=0 ; i < $(arr_count dyn_$name) ; i++ ))
	do
		echo "dyn_$name[$i]: $(arr_at dyn_$name $i)"
		
	done
done

for name in "${array_names[@]}"
do
	echo "Dumping dyn_$name"
	for n in $(arr_get dyn_$name)
	do
		echo $n
	done
doneCode language: PHP (php)

This final example uses a Bash 4.3 feature called ‘nameref’ which allows you to reference a variable by the variable name which is much safer than using ‘eval’, however; depending on the system being used Bash 4.3 may not be available.

# Dynamically create an array by name
function arr() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -g -a $1=\(\)	
}

# Insert incrementing by incrementing index eg. array+=(data)
function arr_insert() {	
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	declare -n r=$1
	r[${#r[@]}]=$2
}

# Update an index by position
function arr_set() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	declare -n r=$1	
	r[$2]=$3
}

# Get the array content ${array[@]}
function arr_get() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	declare -n r=$1	
	echo ${r[@]}
}

# Get the value stored at a specific index eg. ${array[0]}  
function arr_at() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable" 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	[[ ! "$2" =~ ^(0|[-]?[1-9]+[0-9]*)$ ]] && { echo "Array index must be a number" 1>&2 ; return 1 ; }
	declare -n r=$1	
	local max=${#r[@]}
	# Array has items and index is in range
	if [[ $max -gt 0 && $i -ge 0 && $i -lt $max ]]
	then 
		echo ${r[$2]}
	fi
}

# Get the value stored at a specific index eg. ${array[0]}  
function arr_count() {
	[[ ! "$1" =~ ^[a-zA-Z_]+[a-zA-Z0-9_]*$ ]] && { echo "Invalid bash variable " 1>&2 ; return 1 ; }
	declare -p "$1" > /dev/null 2>&1
	[[ $? -eq 1 ]] && { echo "Bash variable [${1}] doesn't exist" 1>&2 ; return 1 ; }
	declare -n r=$1
	echo ${#r[@]}
}



array_names=(bob jane dick)

for name in "${array_names[@]}"
do
	arr dyn_$name
done

echo "Arrays Created"
declare -a | grep "a dyn_"

# Insert three items per array
for name in "${array_names[@]}"
do
	echo "Inserting dyn_$name abc"
	arr_insert dyn_$name "abc"
	echo "Inserting dyn_$name def"
	arr_insert dyn_$name "def"
	echo "Inserting dyn_$name ghi"
	arr_insert dyn_$name "ghi"
done

for name in "${array_names[@]}"
do
	echo "Setting dyn_$name[0]=first"
	arr_set dyn_$name 0 "first"
	echo "Setting dyn_$name[2]=third"
	arr_set dyn_$name 2 "third"
done 

declare -a | grep 'a dyn_'

for name in "${array_names[@]}"
do
	arr_get dyn_$name
done


for name in "${array_names[@]}"
do
	echo "Dumping dyn_$name by index"
	# Print by index
	for (( i=0 ; i < $(arr_count dyn_$name) ; i++ ))
	do
		echo "dyn_$name[$i]: $(arr_at dyn_$name $i)"
		
	done
done

for name in "${array_names[@]}"
do
	echo "Dumping dyn_$name"
	for n in $(arr_get dyn_$name)
	do
		echo $n
	done
done
Code language: PHP (php)

In conclusion, I hope this will help people get their heads around dynamically named variables and dynamically named arrays in Bash. Although these data structures are well known and simple; trying to implement them to be defined on the fly can be frustrating especially without the added ‘nameref’ capabilities of newer versions of Bash.

Getting Bashed by Dynamic Arrays

Recent Posts

Archives

Categories