The irony here is that this is a long article in which I most certainly type too much. This is far more an overview article than anything, and there will be followups detailing the covered sections at a later date.

Noted that I’ll try and include books I’ve read on some of the below that I’ve found handy. Know of another? Leave a comment!

Learn your Shell

http://linuxcommand.org/tlcl.php

Chances are high you’ve been repeating a lot of commands on your shell, especially around git and history based items. After a while all of those characters can add up, time to cut them down to size.

Noted that I mentioned some of this in an earlier article: http://baweaver.dev/blog/2013/09/29/getting-cozy-with-the-command-line/

If you notice yourself typing something more than once, or typing a string of commands you tend to forget, it’s time to break out some shell scripting and knock things down to size.

ZSH

If you haven’t checked it out yet, ZSH is loaded with aliases and extra power from the start. I won’t cover the list of features, but the ones we should be concerned with are (and some exist in BASH):

• Aliasing
• Tab Completion
• History
• Globbing

Aliasing

(note: Oh-My-ZSH has a lot of this built in: https://github.com/robbyrussell/oh-my-zsh)

How often do you find yourself typing in git add or git commit -m or other items? You can alias those into a few characters, saving a lot of typing:

1
2
3
4

alias g='git'
alias gcm='git commit -m'
alias gcb='git checkout -b'
alias gpo='git push origin'

Now what’s the difference here between that and Bash? ZSH supports global aliases which can be used anywhere in a command. Let’s say you want to keep a log of a statement:

1

alias -g LOG="| tee -a ~/log.txt"

Functions

Functions, much like any other language, can be used to combine actions. Sometimes a quick function in your shell is all you need.

How about getting your current branch name for a commit message?

1
2
3
4
5
6
7
8
9
10
11

function branch_name() { git rev-parse --abbrev-ref HEAD }

git push origin `branch_name`

# Though you could also just:
git push origin HEAD

# Though what if you want your commit messages prefixed with your task?
#
# ex: ABC-123-my-branch-name
function branch_prefix() { branch_name | cut -d'-' -f1,2 }

I tend to use this a lot for grep, less, and other common shell functions in my workflow. It’s really handy when it’s some heinous AWK or SED line I don’t want to remember.

Tab Completion

While this works in Bash with some extensions, it comes built into ZSH. Even more comes up when you have Oh-My-ZSH which uses Compleat: https://github.com/mbrubeck/compleat

If you’re like me and you prefix your git branches with tags, you can autocomplete against that if you happen to misplace a branch.

Learn your Editor

Your editor has features designed to save time as well. While autocompletion comes to mind, I personally find it tedious and not nearly as powerful as other features such as snippets and macros.

Sublime

https://sublimetextbook.com/

The first considerations in sublime should be those mentioned on the front page, such as column selection and multi-select for words. It’s worth it to read the documentation there as there are several features that will be immediately usable.

Plugins

There’s a sublime plugin for pretty well everything, including that documentation you hate to write by hand and can never remember the syntax for.

Snippets

Sublime comes built with the concept of snippets, letting you define blocks of code with interpolatable tags:

1
2
3

function ${1:myFunction} (${2:args}) {
  ${3:return;}
}

These can be bound to language specific contexts, preventing overlaps for potentially the same names (jasmine vs rspec snippets anyone?)

Macros

http://docs.sublimetext.info/en/latest/extensibility/macros.html

These are going to look very familiar if you’ve been using vim, at least in terms of key commands.

A macro is a series of actions that can be replayed at a later time, even bound to a key combination.

Catch yourself correcting 4 space indentation to 2 space? You can macro that!

An evil left-bracer got a hold of your files? You can macro that!

Someone is writing Java on your team and you want to get rid of it for Scala? You can macro that one too, but a bit more hackery and a priest to contain the evil during the exorcism will be required.

Vim

https://pragprog.com/book/dnvim/practical-vim

Much of the same general features in Sublime are available in Vim with some extension, including substantially more powerful macro and snippets features. Sublime just happens to come pre-baked with simpler sane defaults.

Shell out

Vim can use the command system to execute whatever you want from the shell, learning to use this will be of extreme benefit. You can even go as far as having your own scripts directory for generating more code on the fly.

Ulti-Snips

https://github.com/SirVer/ultisnips

You remember how sublime snippets can interpolate values for you? Ultisnips takes it a step further by taking those interpolated values and using them to generate more.

Why is this useful? Think initializers and documentation skeletons. Learn a bit of Python and you can be off with substantially more dynamic snippets.

What about x IDE?

IDEs are designed to cater to a wide base, and more times than not I find that assumption to make it very annoying to work with. Instead, editors like Vim and Emacs allow me to build things from the ground up, catered specifically to my style of programming.

It should come as little surprise that you tend to remember shortcuts that you yourself make as opposed to memorizing a list of commands and keyboard shortcuts.

Then why do I use Sublime on occasion you may ask? Sublime is far less likely to cause someone to incite physical violence against my person when pair programming than a modded-out instance of Vim with remapped keys everywhere.

That, and Sublime is quite frankly a much better editor for people starting out.

But emacs!

My religious preferences (vim) prevent me from giving credence to such an eVil editor :P. On a serious note, never saw much of a reason to bother with it as I already knew Vim from SysAdmin work.

Learn to recognize unabstractable duplication

There are a few frameworks out there that have a concept of generators. Two of them are Yeoman for NodeJS and Generators for Rails.

Creating generators that are catered to the style of your team can greatly reduce the time and mistakes made when implementing a new section of code. Even if that code can only accurately generate up to 70% of your shippable code, that’s 70% you know works and has passed style inspections and the like.

Yeoman Generators

http://yeoman.io/authoring/

Yeoman, it seems, has a bit of a sense of humor in that they’ve gone and made a generator-generator to help you make more generators. A bit meta, but quite useful in getting started.

Yeomen generators come with options for making prompts much like a wizard, and the nice thing is that they remember your last responses as the new default options.

Say you like a certain generator but need to get more done, just compose it with another generator to get them to run in tandem.

You could even tie them into your editor using custom made adapters. As long as you respond to IO properly, Yeoman can take care of the rest for you.

Rails Generator

A lot of people I know in the Rails world complain about the viability of scaffolded code, saying it comes no where close to what they intended. Well handily enough you can customize every step of the scaffold process, including the style of models, controllers, and views they generate.

Say you just want a simple searchable and sortable bootstrap table with CRUD operations, maybe make that an Angular or React view? You can do that with generators with some customization. You can even generate the RSpec and Jasmine for it while you’re at it.

Don’t underestimate Rails Generators because they’re abused by newer coders. Thumbing your nose at it is a serious mistake.

Learn to let your code speak for itself

Of course you could make generators for all of your code, define the perfect styles and agree on everything, but what if something else could generate it already? Seems like a waste to ignore.

Fortunately there’s such a concept of generating services code for RESTful APIs, present in a number of frameworks:

• RAML - http://raml.org/
• API Blueprint - https://apiblueprint.org/
• Swagger - http://swagger.io/

In most of these you can treat your API definitions as generators for your client service code. Imagine not having to write services in Angular or other frontend frameworks.

Now if you’re clever, you could even tie this into an inline-api tool like ApiPie to create things in line with your actual API methods giving you both documentation and workable client code at the same time.

Finishing up

The main thing in reducing typing is to never be content running a long string of tasks. Always be looking for areas that can be improved, reduced, or eliminated altogether. Most of this same mentality is already applied to our code, so why not the process leading up to and around our code as well?

Here are a few questions to get you thinking before we start. Given a model Foo which has_many Tags (key, value, foo_id):

• How do we find the count of tags for every Foo?
• How do we find a Foo with multiple matching tags? (name: ‘David Tennant’ AND color: ‘Blue’)

Suddenly ActiveRecord becomes very annoyingly complicated to use, but not to fear! We can still use SQL for all of this.

So now our application looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# Foo: {a: String, b: String, c: String}
# Tag: {key: String, value: Text, foo_id: Integer}
# Foo has_many Tags and Tag belongs_to a Foo

# Seeds

words   = IO.readlines('/usr/share/dict/words').flat_map { |w| w.chomp.downcase }
records = 100.times.map { |i| {a: words.sample, b: words.sample, c: words.sample} }
foos    = Foo.create(records)

tag_seeds = {
  name:  [
    'William Hartnell',
    'Patrick Troughton',
    'Jon Pertwee',
    'Tom Baker',
    'Peter Davison',
    'Colin Baker',
    'Sylvester McCoy',
    'Paul McGann',
    'Chris Eccleston',
    'David Tennant',
    'Matt Smith',
    'Peter Capaldi'
  ],
  place: %w(Tardis Gallifrey Kasterborous Earth),
  color: %w(Red Blue Yellow Green Black White Orange)
}

foos.each { |foo|
  tag_seeds.each { |key, values|
    foo.tags << Tag.create(key: key, value: values.sample)
  }
  foo.save
}

A Path Lesser Traveled

Normally when you start to look into ActiveRecord Queries, you’re going to see some code that looks something like this:

1

Foo.where(a: 1, b: 2, c: 3)

or perhaps you’ll see the normal escaped queries:

1

Foo.where('a = ? AND b = ? AND c = ?', 1, 2, 3)

…but lurking in the documentation you’ll find another way entirely that’s not so often advertised by guides, allowing us the same power as the string based conditionals with what I would argue as a lot more clear way.

1

Foo.where('a = :a AND b = :b AND c = :c', {a: 1, b: 2, c:3})

So why not just use the first variant like any sane developer, you might wonder. Put simply, because this allows us the full leverage of string conditionals with a lot more clarity. Try and do this with the hash syntax:

1
2
3
4

Foo.where(
  'a LIKE :a OR b LIKE :b AND created_at > :date AND length(:c) > 5',
  {a: 'a%', b: 'b%', c: 5, date: 20.days.ago}
)

Counting associations

Say you want the count of tags on a foo, how would you go about it?

1

Foo.joins(:tags).group('foos.id').count('tags.id')

Now the conundrum here is why do we need to use group? Let’s take a look at the generated SQL:

1
2

Foo.joins(:tags).group('foos.id').count('tags.id')
   (0.5ms)  SELECT COUNT(tags.id) AS count_tags_id, foos.id AS foos_id FROM "foos" INNER JOIN "tags" ON "tags"."foo_id" = "foos"."id" GROUP BY foos.id

In order to run a count on an association, we need to aggregate the records into groups that we’ll run the count against. Handy thing is, this allows us to do a few more… interesting things:

1
2
3

[35] pry(main)> Foo.joins(:tags).group('tags.key').count('tags.id')
   (0.6ms)  SELECT COUNT(tags.id) AS count_tags_id, tags.key AS tags_key FROM "foos" INNER JOIN "tags" ON "tags"."foo_id" = "foos"."id" GROUP BY tags.key
=> {"color"=>100, "name"=>100, "place"=>100}

You can run aggregates for different groups, not just the supposedly common case, hence why AR wants you to specify it. In the first case, we’re simply telling it to aggregate the tags based on the id of their parent.

Finding multiple matching tags

I’m going to put a disclaimer here and say that trying to play for Single Table Inheritance hacks like this will cause you a lot more harm than good. Now if your data model is not so friendly and forces you into this, it’s something worth remembering.

Aliased Inner Joins

SQL has a concept for this, but AR currently does not give us this power. Thankfully we have access to find_by_sql:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

search_tags = [
  {key: 'name', value: 'David Tennant'},
  {key: 'color', value: 'Blue'}
]

aliased_tags = search_tags.map.with_index { |tag, i| ["t#{i}", tag] }.to_h

sql_data = aliased_tags.reduce({
  sql: '', data: [], where: {}
}) { |state, (i, tag)|
  state[:sql]  << " INNER JOIN tags AS #{i} ON #{i}.key = ? "
  state[:data] << tag[:key]

  state[:where].merge!(i => {"#{i}_value" => tag[:value]})

  state
}

where_clause = sql_data[:where].reduce({
  sql_fragments: [], data: {}
}) { |state, (i, tag)|
  state[:sql_fragments] << "#{i}.value = :#{tag.keys.first}"
  state[:data].merge!(tag.symbolize_keys)
  state
}

where_sql = Foo.where(
  where_clause[:sql_fragments].join(' AND '),
  where_clause[:data]
).to_sql

select_sql, new_where_sql = where_sql.split('WHERE')
final_sql = select_sql + ' ' + sql_data[:sql] + " WHERE " + where_sql

Foo.find_by_sql([final_sql, *sql_data[:data]])

# The final SQL looks something like this:
SELECT "foos".* FROM "foos"
  INNER JOIN tags AS t0 ON t0.key = 'name'
  INNER JOIN tags AS t1 ON t1.key = 'color'
  WHERE (t0.value = 'David Tennant' AND t1.value = 'Blue')

Now note that this is most certainly not the best way to go about this, suggestions are quite welcome as to better ways to deal with this one.

What we’re doing here is in essence creating an on-the-fly single table inheritance to query against.

Admittedly a better way to do this currently eludes me, and I would recommend against using this on your own solutions.

One can use subqueries to circumvent this type of issue, but the solution will be similar if not slower.

Finishing up

As you can see, this design of ours quickly devolves into madness when querying against at the end of the article. In the next sections, I’ll be covering methods of database design to avoid these issues as much as possible.

For this we’ll be using a model called Foo with the fields a, b, and c. All of the fields are strings with random words chosen from OSX’s built in wordlist:

1
2
3
4

# rails g model foo a b c
words   = IO.readlines('/usr/share/dict/words').flat_map { |w| w.chomp.downcase }
records = 10_000.times.map { |i| {a: words.sample, b: words.sample, c: words.sample} }
Foo.create(records)

Count

How many times have you done this?

1

Model.all.size

The problem with this one is quite simply that it’s retrieving all the records just to get a count. Seem inefficient? It is:

1
2

[3] pry(main)> Foo.all
  Foo Load (33.7ms)  SELECT "foos".* FROM "foos"

Instead, use the count method:

1
2

[4] pry(main)> Foo.count
   (0.2ms)  SELECT COUNT(*) FROM "foos"

Let’s go ahead and blank out the a field for the first thousand or so records. Note that I don’t use first, as that returns an array:

1
2
3

[8] pry(main)> Foo.where('id < 1000').update_all(a: nil)
  SQL (3.5ms)  UPDATE "foos" SET "a" = NULL WHERE (id < 1000)
=> 999

Now how would we get the count of records where a is present? Count takes arguments:

1
2
3

[9] pry(main)> Foo.count(:a)
   (1.5ms)  SELECT COUNT("foos"."a") FROM "foos"
=> 9001

Would you look at that, it’s over 9000!

Group

Let’s say we want to group our records by their length to find out how many words there are for a certain length. Ruby has a built in group_by method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

[28] pry(main)> Foo.all.group_by { |v| v.a.try(:size) }.map { |k,v| [k,v.size] }.to_h
  Foo Load (22.5ms)  SELECT "foos".* FROM "foos"
=> {nil=>999,
 14=>348,
 10=>1246,
 11=>992,
 4=>204,
 8=>1166,
 9=>1262,
 5=>404,
 6=>630,
 7=>856,
 12=>795,
 16=>117,
 13=>576,
 15=>220,
 17=>65,
 3=>51,
 18=>40,
 19=>14,
 2=>4,
 20=>6,
 1=>2,
 22=>2,
 21=>1}

That all should be enough of a trigger to start looking for an aggregate method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

[29] pry(main)> Foo.group('length(a)').count
   (9.7ms)  SELECT COUNT(*) AS count_all, length(a) AS length_a FROM "foos" GROUP BY length(a)
=> {nil=>999,
 1=>2,
 2=>4,
 3=>51,
 4=>204,
 5=>404,
 6=>630,
 7=>856,
 8=>1166,
 9=>1262,
 10=>1246,
 11=>992,
 12=>795,
 13=>576,
 14=>348,
 15=>220,
 16=>117,
 17=>65,
 18=>40,
 19=>14,
 20=>6,
 21=>1,
 22=>2}

SQL functions are perfectly valid in this context, and quite helpful as well. Just using a column name in group, we can group by similar values as well.

Pluck

Pluck doesn’t just get certain columns from a database, it can also be used for SQL functions. Let’s say we want a list of what length of words we have:

1
2
3

[35] pry(main)> Foo.pluck('DISTINCT length(a)')
   (3.3ms)  SELECT DISTINCT length(a) FROM "foos"
=> [nil, 14, 10, 11, 4, 8, 9, 5, 6, 7, 12, 16, 13, 15, 17, 3, 18, 19, 2, 20, 1, 22, 21]

How about the average length of our a column?

1
2
3

[36] pry(main)> Foo.pluck('avg(length(a))')
   (2.2ms)  SELECT avg(length(a)) FROM "foos"
=> [9.574158426841462]

Noted you can use the average(:a) function here as well:

1
2
3

[39] pry(main)> Foo.average('length(a)')
   (2.9ms)  SELECT AVG(length(a)) FROM "foos"
=> #<BigDecimal:7fcf75efd2f0,'0.9574158426 84146E1',27(36)>

…but what you cannot do with average, min, max, and other calculation functions is this useful tidbit:

1
2
3

[40] pry(main)> Foo.pluck('avg(length(a))', 'max(length(a))', 'min(length(a))', 'count(a)')
   (5.4ms)  SELECT avg(length(a)), max(length(a)), min(length(a)), count(a) FROM "foos"
=> [[9.574158426841462, 22, 1, 9001]]

That one, without aggregate functions, is likely to take quite a while indeed.

Calculations

There are some other common functions that may well come in handy if you only happen to need one value:

1
2
3
4
5
6
7

[42] pry(main)> Foo.minimum('length(a)')
   (2.4ms)  SELECT MIN(length(a)) FROM "foos"
=> 1

[43] pry(main)> Foo.maximum('length(a)')
   (2.2ms)  SELECT MAX(length(a)) FROM "foos"
=> 22

Where

Not an aggregate per-se, but using a where clause can still use SQL functions. Say you only want records with an a field longer than 10 characters:

1
2
3

[44] pry(main)> Foo.where('length(a) > 10').count
   (2.1ms)  SELECT COUNT(*) FROM "foos" WHERE (length(a) > 10)
=> 3176

Maybe the count isn’t what you’re after. Perhaps you want the ids instead?:

1
2
3
4
5
6

[45] pry(main)> Foo.where('length(a) > 10').ids
   (5.0ms)  SELECT "foos"."id" FROM "foos" WHERE (length(a) > 10)
=> [1000,
 1002,
 1004,
 # ...

Never underestimate the value of being familiar with basic functions in SQL, as they’ll save your database a lot of headaches.

Finishing up

While a strong knowledge of SQL is not always necessary for Rails development, it will most certainly improve your code and your performance. Not everything has to fit into hash arguments for a where clause.

This post will outline the beginnings of the madness that led to Clairvoyant as well as some of the details of how things are planned to be implemented.

Code as Data

The LISPers among you will notice a very common theme throughout this post. That theme is quite simply that I’m taking a ruby file and treating it as data for an entirely different parser.

The DSL is already there, the data set, the question becomes what can we divine from what we have with reasonable certainty?

Logic Languages

Along with inspirations from LISP, we’re drawing pretty from Logical languages such as Prolog. A logic program is a statement of facts used to derive an answer to a question:

1
2
3
4
5
6
7
8
9
10

{http://www.csse.monash.edu.au/~lloyd/tildeLogic/Prolog.toy/Examples/}
witch(X)  <= burns(X) and female(X).
burns(X)  <= wooden(X).
wooden(X) <= floats(X).
floats(X) <= sameweight(duck, X).

female(girl).          {by observation}
sameweight(duck,girl). {by experiment }

? witch(girl). {Now we ask it}

Now I don’t pretend to be an expert in Prolog, or even really particularly any good at it. Given that, it still reminds me of something:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

describe Witch do
  describe '#burns' do
    it 'burns' do
      expect(subject.burns).to eq(true)
    end
  end

  describe '#wooden' do
    it 'is made of wood' do
      expect(subject.wooden).to eq(true)
    end
  end

  # ...
end

So if RSPEC looks like it’s already making assertions about the nature of our program, what happens if we treat it like a logic language?

Repurposing a DSL

The thing about Ruby is it’s incredibly flexible. The DSL from RSPEC can just as easily be hijacked and run in another compiler with this one simple trick (and I swear I won’t clickbait):

1
2
3
4
5
6

class MyParser
  def initialize(file_lines)
    @descriptions = []
    self.class_eval(file_lines)
  end
end

Now what have we done? We’ve evaluated the entirety of the loaded file in the context of our class. What happens if we redefine describe inside of there?

1
2
3
4
5
6
7
8
9

def describe(description, &block)
  @descriptions << description
  block.call
end

# Now when it hits 'Witch', it returns a symbol of the name instead
def const_missing(name)
  name
end

We can capture the entirety of the describe blocks, or for that matter anything else we want. As long as the spec file isn’t using ::RSpec.describe we can hijack whatever we want. If it does, it just makes it mildly more annoying to reason about.

This is effectively the current state of Clairvoyant. You can take a very basic spec file and generate a skeleton of it such that:

1
2
3
4
5
6
7

describe Foo do
  describe '#bar' do
    it 'does something magical' do
      expect(subject.bar).to eq(5)
    end
  end
end

Will generate:

1
2
3
4
5
6
7
8

class Foo
  # It does something magical
  #
  # @return [Integer]
  def bar
    # Code goes here later
  end
end

Granted that’s not all that impressive at this point, but beyond this stage we’re going to find some very interesting problems. This brings me to my next section.

Theoreticals

Generating a skeleton is easy enough, and still very useful in its own right. Actually writing code from expectations and matchers that could be near infinite in number and complexity? That becomes a whole different story very quickly. These are theoretical musings of the future nature of Clairvoyant as I see it.

The nature of the ‘it’ block

Past the description, we’re defining what the logic of the program is. From here we can infer a good number of relevant details:

1
2
3

it 'does this' do
  expect(method(a,b,c).last).to eq(5)
end

The description string of the method can be used for documentation and a nifty method description in some cases.

The actual expectation call tells us the name of the method, and potentially anything that we can call after it. In the above example we know that last can be called on the result of our method and the arity of the method can be 3. We can also guess that the method is some form of Enumerable, dropping possible options for output substantially. Given that we only have an integer here, we can make a reasonable statement that the return value of method is Array[Integer]

We have facts to work with here, and the more it methods we have inside of a describe, the more we can divine from given facts of the method. Say another test called keys on our methods return, now we can reasonably guess it’s a Hash or close derivative.

That’s well and good, but a lot of ruby methods tend to be very conditional in nature. They could return different things dependent on a context. Luckily we have just such a method we can hijack!

Contextual contexts abound

Say we find our method doing strange things dependent on what the context is. We can possibly even derive a conditional from a well laid out context description:

1
2
3

context 'When value is even' do
  # ...
end

We have a potentially bindable name in value, and a proper context test to throw against with is even. Of course at this point it’s going to be a lot more difficult to glean this information and will be heavily dependent on the robustness of a tokenization library and parser.

This becomes substantially more difficult to reason about, because now we’re trying to tell people how to write their tests instead of divining information from what’s already there. That may be fine for new code but can be incredibly tedious to make behave properly.

Expectational matchers

Matchers provide an even more interesting challenge, especially factoring in user defined options. We can make some reasonable assertions based on raise error to give some error handling for dynamic methods, but that again becomes dependent on context blocks being clear enough to grok.

Meta-testing

Given that we have all of that figured out, the next fun part is proving whether or not what we did even works right. At this point we can run our generated code through the RSPEC again as the core team intended to see if we made it pass. If we did, great! That’s the easy case if we’ve already gotten this far. If we haven’t on the other hand it opens up a whole different can of worms.

Meta-Meta-testing

So maybe it didn’t pass. Hey! It gave us data back to use to further refine and polish our solutions. We can use that to (hopefully) get them to pass on the next round! At this point the failed tests would be ported back and we could do a few things at this point:

Fail the method and leave a comment for the user or Attempt to repair the method to make the test pass

The first would be far more practical if we’ve gotten this far, but hey, we’re in theory land. Let’s push our luck a bit more here.

Meta-.*-testing

At this stage we would be throwing code back and forth until something works, a very brute force solution to hoping we hit the sweet spot. In something that could only be compared to the quandry of monkies writing Shakespear, we might squeeze just a bit more code out of there.

Though honestly, halting problem is just a euphemism for being dull, let’s have more fun!

S-Expression analysis

So we can get a hold of a lot of your application code as well right? Let’s not limit that. Let’s grab as much ruby code as we can stuff into memory and try and find patterns between their RSPEC code and application code. Machine learning and deep analysis can be applied to more acurately divine intended code based on community behaviors (though I will explicitly prune out you maniacs who use globals like candy.)

Throw it in a Spark cluster and let the thing roar. We’re deep into AI land of making some very interesting code generation black magic happen, and probably well beyond anything that’s been attempted up to this point.

I have no qualms saying this is well beyond me, but it sounds like a blast to try anyways.

You’re out of your mind

It wouldn’t be the first time I’ve been told this, and certainly won’t be the last. This is a personal project and a great deal of fun in learning Ruby internals along the way. Maybe one day this will be a fully functional project that can magically make your wildest dreams come true, or maybe not.

Really, when it gets down to it, that’s the fun of it. The potentials here are limitless, the problem hard, and the code plentiful. That’s the best type of problem to poke at. It’d be no fun if I knew entirely what I was doing.

Check out Clairvoyant, leave me a comment, let me know what you think!

Getting Spark

The first step to running Spark is to get a standalone instance to play with on our machines.

Go to the Spark homepage: https://spark.apache.org/downloads.html

We’ll be using version 1.4.0. Select that version from releases, and select Pre-built for Hadoop 2.6 and later (unless you currently have another Hadoop / HDFS instance at a different version.)

Go ahead and download / unpack that into the directory of your choice, and cd into it.

Getting our wordlist

We’ll be using an english wordlist from SIL for the following exercises. Make sure to save wordsEn.txt somewhere where you can load it later.

Spark REPL

The last tutorial mentioned the concept of a REPL as a way to play with code interactively. Handy enough, Spark implemented its own REPL over Scala and Python (and not Java.)

For Scala that would be bin/spark-shell

For Python, it’s bin/pyspark

You should see something like this (snipped for length):

1
2
3
4
5
6
7
8
9
10

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.3.1
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_31)
Type in expressions to have them evaluated.
Type :help for more information.

or this:

1
2
3
4
5
6
7
8
9

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.3.1
      /_/

Using Python version 2.7.5 (default, Mar  9 2014 22:15:05)
SparkContext available as sc, HiveContext available as sqlContext.

There will be a considerable amount of other debugging and logging statements than that, but for the point of this those will do as things to look for.

Spark Context

In the Spark shell, we’re given the entirety of the Spark library as sc to interact with. We can use that to load in our text file:

1
2

scala> val wordList = sc.textFile("/Users/lemur/dev/wordlist/wordsEn.txt")
// ...debugger output

1
2

>>> wordList = sc.textFile("/Users/lemur/dev/wordlist/wordsEn.txt")
# ...debugger output

WARNING - Remember last time when I mentioned Spark was Lazy? If you type that path in wrong, it’s not going to tell you anything until you try and run commands on it. This is the same for a lot of functions in Spark, you won’t know it’s broken until you run it.

Now we have our files loaded into memory to do some experimentation with as RDDs (Resilient Distributed Datasets), Spark’s abstraction for distributed data.

Let’s try a basic one to start, how many lines are in the file? (I’m going to be trimming output so we don’t fill the page with debugger info)

1
2
3

scala> wordList.count()
// ...debugger output
res0: Long = 109583

1
2
3

>>> wordList.count()
# ...debugger output
109583

With that you’ve just run a Spark job. Simple as that, and not much different than how you’d interact with anything else.

Starts with

Now, since this is a dictionary, each word is in there once. That makes a wordcount a bit pointless, so instead let’s get a list of what letters they start with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

scala> wordList.filter(_ != "").map(word => (word(0), 1)).reduceByKey(_+_).foreach(println)

(w,2714)
(s,12108)
(e,4494)
(a,6541)
(k,964)
(i,4382)
(y,370)
(u,3312)
(o,2966)
(q,577)
(g,3594)
(d,6694)
(z,265)
(m,5806)
(c,10324)
(p,8448)
(x,79)
(t,5530)
(b,6280)
(h,3920)
(n,2475)
(f,4701)
(j,1046)
(v,1825)
(r,6804)
(l,3363)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

letterCounts = wordList \
  .filter(lambda w: w != "") \
  .map(lambda w: (w[0], 1)) \
  .reduceByKey(lambda a,b: a + b) \
  .collect() # Force the result to run

>>> for count in letterCounts:
...   print count
...
(u'a', 6541)
(u'c', 10324)
(u'e', 4494)
(u'g', 3594)
(u'i', 4382)
(u'k', 964)
(u'm', 5806)
(u'o', 2966)
(u'q', 577)
(u's', 12108)
(u'u', 3312)
(u'w', 2714)
(u'y', 370)
(u'b', 6280)
(u'd', 6694)
(u'f', 4701)
(u'h', 3920)
(u'j', 1046)
(u'l', 3363)
(u'n', 2475)
(u'p', 8448)
(u'r', 6804)
(u't', 5530)
(u'v', 1825)
(u'x', 79)
(u'z', 265)

Spark SQL

On occasion we’ll have the niceties of structured data such as JSON, and Spark has just the way to deal with it using Spark SQL.

WARNING - Spark guide has been quoted as saying:

Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

…and it will crash if you pass it actually valid JSON. If any reader knows the reasoning behind this particularly confounding piece of work, I’d love to know.

We’ll be using fake people data: https://gist.githubusercontent.com/baweaver/b6460bb96feff1faeb78/raw/4c9b46be165725d041ff47bdc042c6a4880c1877/people.json (right click to save)

Let’s go ahead and load it up using the sqlContext:

1
2
3
4
5

scala> val people = sqlContext.jsonFile("/Users/lemur/dev/wordlist/people.json")
people: org.apache.spark.sql.DataFrame = [_id: string, address: string, age: bigint, balance: double, company: string, email: string, eyeColor: string, gender: string, guid: string, index: bigint, isActive: boolean, latitude: double, longitude: double, name: string, phone: string, picture: string, registered: string]

// Make SURE to register it as a table
scala> people.registerTempTable("people")

1
2
3
4
5
6
7

>>> people = sqlContext.jsonFile("/Users/lemur/dev/wordlist/people.json")

>>> people
DataFrame[_id: string, address: string, age: bigint, balance: double, company: string, email: string, eyeColor: string, gender: string, guid: string, index: bigint, isActive: boolean, latitude: double, longitude: double, name: string, phone: string, picture: string, registered: string]

# Make SURE to register it as a table
>>> people.registerTempTable("people")

Let’s start with something fairly basic on the SQL, getting the index of people who are inactive with a balance greater than $2000:

1
2
3
4
5
6
7
8
9

// Note I'm calling on SQL Context here
scala> sqlContext.sql("""
     |   SELECT index
     |   FROM people
     |   WHERE isActive == false AND
     |         balance > 2000.00
     | """).count()

res1: Long = 75

1
2
3
4
5
6
7
8

>>> sqlContext.sql("""
...   SELECT index
...   FROM people
...   WHERE isActive == false AND
...         balance > 2000.00
... """).count()

75

Triple quotes are a life saver when making larger SQL-like strings.

Like SQL, you can join, count, group, and various other operations all in a big data context. It’s a shame it won’t play nicely with actual JSON, but the features are handy nonetheless.

Further reading

Spark MLLib - Statistics

Spark even comes with its own Machine Learning libraries, but for the sake of brevity we’re only going to look into some of the basic statistical options. Later tutorials will address this in some depth.

We’ll be looking into the column stats of our wordList from earlier:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Make SURE to import it
scala> import org.apache.spark.mllib.stat.Statistics
scala> import org.apache.spark.mllib.linalg.Vectors

scala> val wordList = sc.textFile("/Users/lemur/dev/wordlist/wordsEn.txt")

scala> val wordLengths = wordList.map(w => Vectors.dense(w.length))
wordLengths: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[6] at map at <console>:32

scala> val summaryStatistics = Statistics.colStats(wordLengths)
summaryStatistics: org.apache.spark.mllib.stat.MultivariateStatisticalSummary = org.apache.spark.mllib.stat.MultivariateOnlineSummarizer@4377e40a

// Let's take a look inside shall we?
scala> summaryStatistics.mean
res22: org.apache.spark.mllib.linalg.Vector = [8.533905806557591]

scala> summaryStatistics.max
res23: org.apache.spark.mllib.linalg.Vector = [28.0]

scala> summaryStatistics.min
res24: org.apache.spark.mllib.linalg.Vector = [0.0]

scala> summaryStatistics.variance
res25: org.apache.spark.mllib.linalg.Vector = [6.448337984119102]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Make SURE to import it
>>> from pyspark.mllib.stat import Statistics

>>> wordList = sc.textFile("/Users/lemur/dev/wordlist/wordsEn.txt")

# Python will take a standard list in
>>> wordLengths = wordList.map(lambda w: [len(w)])

>>> summaryStatistics = Statistics.colStats(wordLengths)

# Let's take a look inside shall we?
>>> summaryStatistics.mean()
array([ 8.53390581])

>>> summaryStatistics.max()
array([ 28.])

>>> summaryStatistics.min()
array([ 0.])

>>> summaryStatistics.variance()
array([ 6.44833798])

Further reading

Wrapping Up

We’ve taken a cursory look at some of the features and basic operations of Spark. Here’s the question though, what do you as readers want to know more about? Vote on Strawpoll to let me know: http://strawpoll.me/4701594

Think of it as a choose your own adventure of sorts. I’ll be writing about all of the above in more detail, but in the order you want to see it happen.

The goal of this post is to get you up to speed in the very basics of Functional Programming as they’ll later relate to Spark. I’ll be primarily covering Scala with Python alternate versions.

What about Java 8?

I do not intend to cover Java in this tutorial or any other. MapReduce is a concept based in Functional Programming, and you would be doing yourself a great disservice by trying to shoehorn Java into that role, including Java 8.

You might wonder how bad it could possibly be, perhaps I’m just biased. I would direct you to look at the Hadoop word count example and see the horrors of allowing Java patterns and card carrying GoF members to pretend they’re programming functionally:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Counter;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.util.StringUtils;

public class WordCount2 {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    static enum CountersEnum { INPUT_WORDS }

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    private boolean caseSensitive;
    private Set<String> patternsToSkip = new HashSet<String>();

    private Configuration conf;
    private BufferedReader fis;

    @Override
    public void setup(Context context) throws IOException,
        InterruptedException {
      conf = context.getConfiguration();
      caseSensitive = conf.getBoolean("wordcount.case.sensitive", true);
      if (conf.getBoolean("wordcount.skip.patterns", true)) {
        URI[] patternsURIs = Job.getInstance(conf).getCacheFiles();
        for (URI patternsURI : patternsURIs) {
          Path patternsPath = new Path(patternsURI.getPath());
          String patternsFileName = patternsPath.getName().toString();
          parseSkipFile(patternsFileName);
        }
      }
    }

    private void parseSkipFile(String fileName) {
      try {
        fis = new BufferedReader(new FileReader(fileName));
        String pattern = null;
        while ((pattern = fis.readLine()) != null) {
          patternsToSkip.add(pattern);
        }
      } catch (IOException ioe) {
        System.err.println("Caught exception while parsing the cached file '"
            + StringUtils.stringifyException(ioe));
      }
    }

    @Override
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      String line = (caseSensitive) ?
          value.toString() : value.toString().toLowerCase();
      for (String pattern : patternsToSkip) {
        line = line.replaceAll(pattern, "");
      }
      StringTokenizer itr = new StringTokenizer(line);
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
        Counter counter = context.getCounter(CountersEnum.class.getName(),
            CountersEnum.INPUT_WORDS.toString());
        counter.increment(1);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    GenericOptionsParser optionParser = new GenericOptionsParser(conf, args);
    String[] remainingArgs = optionParser.getRemainingArgs();
    if (!(remainingArgs.length != 2 | | remainingArgs.length != 4)) {
      System.err.println("Usage: wordcount <in> <out> [-skip skipPatternFile]");
      System.exit(2);
    }
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount2.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    List<String> otherArgs = new ArrayList<String>();
    for (int i=0; i < remainingArgs.length; ++i) {
      if ("-skip".equals(remainingArgs[i])) {
        job.addCacheFile(new Path(remainingArgs[++i]).toUri());
        job.getConfiguration().setBoolean("wordcount.skip.patterns", true);
      } else {
        otherArgs.add(remainingArgs[i]);
      }
    }
    FileInputFormat.addInputPath(job, new Path(otherArgs.get(0)));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs.get(1)));

    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

That’s just a word count example, imagine the headaches of anything remotely complex in that paradigm and you’ll do the same as I did and swear off Hadoop.

Are there workarounds for it? Yes. You can also put a nice coat of paint on an old rusty car. It’ll look nicer but it’s not fooling anyone what’s under the hood.

Spark can do better

Remember that word count example? Scala and Spark do it substantially better:

1
2
3
4
5

val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

Even Python leaves it in the dust:

1
2
3
4
5

text_file = spark.textFile("hdfs://...")
counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("hdfs://...")

You don’t have to spend five seconds scrolling to get through that one. The point is that by defining a mapreduce task in terms of functions, we only need to tell Spark what actions it’s taking on the data. We’ll get to what all this means later.

Why Scala over Python?

I advocate the usage of Scala in general for Big Data problems over Python. The reasoning is that Scala is a Statically typed language, surprisingly moreso than even Java (we’ll cover that in a moment.) Spark was also written in Scala, meaning its DSL is going to be very familiar if you’re any grade of Scala programmer.

Python is a great language, don’t get me wrong, but it’s not fully functional. You’ll see why that’s a big deal in a moment here. That, and I don’t like typing lambda all the time.

Basics of Functional Programming

So what is Functional Programming, besides the most thrown around concept in modern days? Quite simply it’s a program built up from Functions instead of Objects.

A little bit more into it, Functional Programming embraces a few interesting ideals:

• The REPL - That’s Read Evaluate Print Loop, a program for running code inline much like a Unix Shell
• Mutation is forbidden - All variables are final
• Functional purity - If you pass A into a function, you’re always getting B back
• Nil is dead - Null pointer exceptions begone!
• Programs are composed of functions - Think writing a program on terms of verbs instead of nouns
• Functions are first class citizens - You can pass functions as arguments, and even return them
• Laziness is useful - Functions and values that don’t evaluate until they’re called

We’ll be getting into each of those and why they’re relevant to Spark here in a moment.

Before we get too far

You’re going to want to get Scala or Python installed so we can get at their REPLs. When you have them installed, drop into your terminal shell and type in either scala or python to drop into a REPL. Give it a swing real quick:

1
2
3
4
5
6

Welcome to Scala version 2.11.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_31).
Type in expressions to have them evaluated.
Type :help for more information.

scala> 5 + 5
res0: Int = 10

1
2
3
4
5

Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 5 + 5
10

In terms of Functional Programming, and later Spark, the REPL will quickly become your best friend.

A Function

So what’s a function? Let’s give the REPL a whirl:

1
2
3
4
5

scala> def add2(x:Int) = x + 2
add2: (x: Int)Int

scala> add2(3)
res1: Int = 5

Fair warning that Python wants you to put a blank line before it assumes you’re done typing

1
2
3
4

>>> def add2(x): return x + 2
...
>>> add2(3)
5

Notice something interesting about Scala there? There’s no need for a return. It’s implied that the last statement in a function is the return value. You’ll also notice that the Scala REPL guessed that we’re going to return an Integer as well, as per the method signature.

Now what’s in a method signature? It’s a contract, a guarantee of a return type. This brings us to our next concept

Goodbye to Nil

Now remember when I said that Scala was more statically typed than Java? Try to give add2 a nil and see what happens in both languages:

1
2
3
4
5

scala> add2(None)
<console>:9: error: type mismatch;
 found   : None.type
 required: Int
              add2(None)

1
2
3
4
5

>>> add2(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in add2
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

But that’s not nil! That’s something called None!

So what’s the difference? Put briefly, Scala and Python both do not have a concept of nil which is a very very good thing for us.

Unless we explicitly tell Scala it can take a None, it will always throw a type error. So how do we define something that might take a value or might not? That’s what we have Option for:

1
2
3
4
5
6
7
8

scala> def add2Maybe(x:Option[Int]) = x.getOrElse(0) + 2
add2Maybe: (x: Option[Int])Int

scala> add2Maybe(Some(2))
res6: Int = 4

scala> add2Maybe(None)
res7: Int = 2

Python does not have this concept, it only replaces nil with None.

Higher Order Functions and Map

One of the most powerful concepts in Functional Programming is the ability to pass functions as arguments. By doing this we’re afforded a great deal of flexibility in defining abstract interfaces for basic operations.

WARNING Python users, normally you’re going to want to use List Comprehensions for this type of thing. Since you’re going to be applying this to Spark, it’s necessary to know these types of functions.

Take map for instance, a function that applies a function to a list. This will be confusing for first timers in this territory, so stick with me for a bit here:

1

scala> List(1,2,3,4).map { i => i * 2 }

1

>>> map(lambda x: x * 2, [1,2,3,4])

Now what do you suppose those two do? We’re passing in a function that takes an argument x and returns x * 2. We’re applying that function to each element in the list, so let’s step through this in Scala:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// val is short for value, or an immutable variable
scala> val myList = List(1,2,3,4)
myList: List[Int] = List(1, 2, 3, 4)

scala> myList.map { i => i * 2 }

// First iteration:  i is 1, returns 2
// Second iteration: i is 2, returns 4
// Third iteration:  i is 3, returns 6
// Fourth iteration: i is 4, returns 8
// ...and now we have a new list returned:
res8: List[Int] = List(2, 4, 6, 8)

// You remember I said immutable? What's myList right now?
scala> myList
res9: List[Int] = List(1, 2, 3, 4)

So not only did we double each element in the list, but our original list is untouched. Given this, we can transform myList however we want and it’ll never change the value of it. Now if we want that result for something, we can always create a new val to save it.

An aside, Scala is very good about trying to simplify things when it can:

1
2

scala> List(1,2,3,4) map (_ * 2)
res10: List[Int] = List(2, 4, 6, 8)

We don’t really need the dot there, and whenever something only takes one parameter Scala will be more than happy to take an underscore to shorten it up for us. While this may seem obscuring to some, it’s a very common pattern in Scala. Best to understand what it’s doing because no amount of rudimentary googling is going to turn that up without some fidgeting, but such are operator and syntactic sugar searches.

Higher Order Functions - Filter

The next function on our list is filter, which takes a function and applies it to each element of a list looking for elements where the result is true:

1
2

scala> List(1,2,3,4) filter (_ > 2)
res10: List[Int] = List(3, 4)

1
2

>>> filter(lambda x: x > 2, [1,2,3,4])
[3, 4]

Higher Order Functions - Reduce

This one is going to be a bit trickier, as it takes a function with two arguments: an accumulator and a value. It reduces a list of elements into one element. Now why would you want such a function? Think of something such as a sum. I’ll be using longhand here as this is one of the harder first functions to really understand:

1
2

scala> List(1,2,3,4).reduce { (accumulator, i) => accumulator + i }
res11: Int = 10

1
2

>>> reduce(lambda accumulator, i: accumulator + i, [1,2,3,4])
10

But how did that work? Let’s step through the logic here in Scala:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

scala> List(1,2,3,4).reduce { (accumulator, i) => accumulator + i }

// In our first iteration, the accumulator is either set to a default value,
// or the head element of the list is used. In this case it's 1

// First iteration - accumulator: 1, i: 2 => 3

// This function returns 3, which is passed in as the next value of the
// accumulator:

// Second iteration - accumulator: 3, i: 3 => 6
// Third iteration  - accumulator: 6, i: 4 => 10

// Now we're out of elements, so reduce returns the accumulator as the result:
res11: Int = 10

Naturally there’s a shorthand for this:

1
2

scala> List(1,2,3,4).reduce(_+_)
res12: Int = 10

An astute reader will notice that we used two underscores here. Scala binds arguments in succession to the underscore, making for a bit more confusion in searching.

Higher Order Functions - Closures

One of the really nifty things about Functions in languages like Scala is that they capture their local environment when they’re defined. What do I mean by that? Let’s take a look:

1
2
3
4
5
6
7
8

scala> def adder(x:Int) = (y:Int) => x + y
adder: (x: Int)Int => Int

scala> val add3 = adder(3)
add3: Int => Int = <function1>

scala> add3(5)
res14: Int = 8

So where did it get 3 from? It remembered the value, or in functional terms it closed over the value when it was defined.

How is this handy? Let’s use map and pass it our function:

1
2

scala> List(1,2,3,4).map(add3)
res15: List[Int] = List(4, 5, 6, 7)

Let’s take it one step further though and just use adder. After all, we might need a bit more flexibility there:

1
2

scala> List(1,2,3,4).map(adder(5))
res17: List[Int] = List(6, 7, 8, 9)

So now we have a function which returns a function that gets used by map. To put it another way, let’s have at filter:

1
2
3
4
5

scala> def divisibleBy(y:Int) = (x:Int) => x % y == 0
divisibleBy: (y: Int)Int => Boolean

scala> List(1,2,3,4).filter(divisibleBy(2))
res18: List[Int] = List(2, 4)

Laziness can be good

You might wonder when a concept like a lazy value might be handy. Let’s say you need an infinite list, or stream, from which you have no clue how many elements you’ll either need or get from it.

How about an infinite stream of Fibonacci numbers, straight from the Scala source code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

scala> val fibs: Stream[BigInt] =
  BigInt(0) #:: BigInt(1) #:: fibs.zip(fibs.tail).map { n => n._1 + n._2 }
fibs: Stream[BigInt] = Stream(0, ?)

scala> fibs.take(10).foreach(println)
0
1
1
2
3
5
8
13
21
34

That’s a lot of code to digest, but it gives us an infinite stream of Fibonacci numbers. Let’s break it apart a bit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// We're creating a new value that's a stream of BigInts
val fibs: Stream[BigInt] =
  // Where the first value is zero, lazily concatenated with
  BigInt(0) #::
  // The second value, which is one, lazily concatenated with
  BigInt(1) #::
  // A function that takes the current fibonnaci numbers,
  // zips them with their tail, and adds those pairs together
  fibs.zip(fibs.tail).map { n => n._1 + n._2 }

// What are zip and tail?
scala> val zipList = List(1,2,3,4)
zipList: List[Int] = List(1, 2, 3, 4)

// Remember head? It gets the first element of our list
scala> zipList.head
res22: Int = 1

// Tail just gets the rest
scala> zipList.tail
res23: List[Int] = List(2, 3, 4)

// It takes two lists and zips them together into tuple pairs
scala> zipList.zip(zipList.tail)
res24: List[(Int, Int)] = List((1,2), (2,3), (3,4))

// Now about that map function:
// map { n => n._1 + n._2 }
//
// That just adds the two elements of the tuple together. In Scala, _n is the
// nth element of the list, non-zero indexed

How is this relevant?

Now that we have all these components, let’s take another look at that wordcount example for Spark:

1
2
3
4
5

val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

Some of those look familiar? Let’s dissect it a bit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// We're reading our document from HDFS, and storing it in textFile
val textFile = spark.textFile("hdfs://...")

// Now we're defining our pipeline - By the way, this is lazy
val counts =
  textFile
    // Flat map is very similar to map, except it flattens the results after it
    // gets them (see flatmap below)
    //
    // What we're doing here is splitting each line by whitespace, and then
    // flattening into one stream of words to go through
    .flatMap(line => line.split(" "))
    // Then we're mapping all those words into a tuple, we'll see why in a
    // second
    .map(word => (word, 1))
    // Reduce by key takes all similar keys and reduces the values with a
    // function, in this case a sum
    .reduceByKey(_ + _)

// Why the tuple and reduce by key? Normally you'd use a groupBy operator here,
// but that does not parallelize cleanly.
//
// What we do to compensate here is make tuples so that we can send specific
// words to different partitions to be reduced

// NOW the pipeline gets called, as we want a value out of it. In this case it
// saves a new text file
counts.saveAsTextFile("hdfs://...")

// Flat Map
scala> List(List(1,2,3), List(2,3,4)).map(list => list.map(adder(2)))
res27: List[List[Int]] = List(List(3, 4, 5), List(4, 5, 6))

scala> List(List(1,2,3), List(2,3,4)).flatMap(list => list.map(adder(2)))
res28: List[Int] = List(3, 4, 5, 4, 5, 6)

Now there’s a lot more to Spark than this, but now you’ve got a grounding by which you can build on. Next we’ll be looking more into Spark specifically.

1
2
3
4
5
6
7

class PeopleController
  def index
    @people = Person.where(name: params[:name]) if params[:name]
    @people = @people.where(birthday: params[:birthday_start]..params[:birthday_end]) if params[:birthday_start] && params[:birthday_end]
    @people = @people.where(sex: params[:sex]) if params[:sex]
  end
end

The horrifying trend will only continue as our demand for searching power grows, which begs the question: How can we tame this mess?

Strong Params

Your first line of defense against this will be using strong params to your advantage. They’re not only for creating objects.

Let’s try something out in the console:

1
2
3
4
5

[1] pry(main)> ActionController::Parameters.new({a: 1, b: 2})
=> {"a"=>1, "b"=>2}
[2] pry(main)> _.permit(:a)
Unpermitted parameter: b
=> {"a"=>1}

So by using permit on our parameters object, we can filter down a hash to only our permitted values. So what if we did something like this?

1
2
3
4
5
6

class PeopleController
  def index
    @people = Person.where(params.permit(:name, :sex))
    @people = @people.where(birthday: params[:birthday_start]..params[:birthday_end]) if params[:birthday_start] && params[:birthday_end]
  end
end

With that we’ve already cleaned out a lot of the cruft of our controller, but what about that last one?

Scoping and Class Methods

We can get rid of it as well, using either scoping or class methods to take care of it for us:

1
2
3
4
5
6
7
8
9

class Person
  # We can go with a scope:
  scope :born_between, -> start, end { where(age: start..end) }

  # ...or a class method:
  def self.born_between(start, end)
    where(age: start..end)
  end
end

Which will let us trim down our controller even a little more here:

1
2
3
4
5

class PeopleController
  def index
    @people = Person.where(params.permit(:name, :sex)).born_between(params[:age_start], params[:age_end])
  end
end

Conditional Scoping

The astute reader will note that the above method is going to fail gloriously should we forget either of those params. We could always drop it to another variable and mutate people, but that’s generally frowned upon and doesn’t normally produce superheroes.

What we can do, however, is introduce a more conditional scoping method. Class methods are, after all, ruby methods. Let’s use them to their potential a bit more:

1
2
3
4
5

class Person
  def self.born_between(start, end = Time.now)
    start ? where(age: start..end) : all
  end
end

By throwing in an all, we can conditionally chain freely.

Like Scoping

The problem is, that name search just isn’t doing it for us. We don’t want to break out solr or trigrams quite yet, but we can use some like queries to make it a bit more flexible:

1
2
3
4
5
6

class PeopleController
  def index
    @people = Person.where(params.permit(:sex)).born_between(params[:age_start], params[:age_end])
    @people = @people.where('name LIKE ?', params[:name]) if params[:name]
  end
end

Though we spend all that time getting rid of postfix if checks, can we do something about this one as well?

1
2
3
4
5
6
7
8
9

class Person
  def self.where_name_like(name)
    name ? where('name LIKE ?', name) : all
  end

  def self.born_between(start_date, end_date = Time.now)
    start_date ? where(age: start_date..end_date) : all
  end
end

That we can:

1
2
3
4
5
6
7
8
9

class PeopleController
  def index
    @people =
      Person
        .where(params.permit(:sex))
        .born_between(params[:age_start], params[:age_end])
        .where_name_like(params[:name])
  end
end

Like that, we’ve eliminated another suffix if.

More advanced filtering

Though most of these examples have been fairly straightforward, there will be times when you have to break out some joins and other operations depending on your parameters. Strong params aren’t going to cut it on those, but class methods just might do the trick.

We have a new model to work with, Post, and with it the following controller: