bobbyratliff

Creating structures that automate things people do every day.

Archive for the ‘Software development’ Category

Cleaning source code with sed

Recently a ticket came across my desk to remove references to Google Plus from a set of email templates our team maintains. With dozens of email templates, I wanted to stretch my command line muscles and see if I could do it all at once.

macOS includes sed, the stream editor, and for this project I’m going to use a very simple set of commands.

Delete lines that match

d is for delete. To use this command, you have a regex, followed by the letter d. It will delete any line that matches.

For example:

sed -i.bak '/googlePlusIcon/d' someFile.txt

Substitute lines

s is for substitute. Think of it like “find and replace” except we are going to find something everywhere. There are 2 parts, a regex to match, and a regex

sed -i.bak 's/googlePlusIcon/twitterIcon/g' someFile.txt

Doing multiple files at once

You can do multiple files at once by combining find with sed. Find syntax is as follows:

find path expression

path is the scope where find will… find things. expression is a set of options to control what we match. For example, -name is to match on the filename.

Another use of expressions is to do some operation. -exec is for executing another command. In this case we want to execute sed. So we type out the sed command from above, except with a few changes. There are curly braces {} which tell find to put the filename of the match in there. There is also the \; which tells find where the -exec expression is ending.

find . -name '*.txt' -exec sed -i.bak 's/googlePlusIcon/twitterIcon/g' {} \;

One quirk about find + sed is that it likes to add newlines to the end of your files. If you are using source control like git, this may cause the diff to be slightly larger than you expected.

Docker cheat sheet

docker run = new container (specify image name)

docker exec = run a command in an existing container

How to turn a docker run command into an interactive shell:

-it --entrypoint=/bin/sh

Note, you have to add that before the name of the image.

Cleanup images

To filter down the images and only clean those matching some-text, use:

docker images | grep some-text | awk '{ print $1 ":" $2}' | xargs docker rmi

Cleanup orphaned images

Sometimes images are orphaned, for example, if you tagged an image as latest, and then built a new image with that same tag. They will show up as <none> in docker images. They can be cleaned up using:

docker rmi $(docker images -f "dangling=true" -q)

List images by size

docker images --format '{{.Size}}\t{{.Repository}}' | sort -hr

What is software architecture?

In my post about Event sourcing and CQRS, I touched on software architecture, but assumed it was already defined. I’d like to dive into that a bit more based on some recent reading.

My definition of software architecture is providing structure that allows productive, effective software development to occur.

Provides structure means having the appropriate technology, practices, culture, and organization in place to allow flourishing. As a software developer I tend to think of the technology (the code) first, but it is certainly not the only thing needed to have effective software architecture. More on that later.

Productive, effective software development means producing software that meets the needs of the business. There are 2 ways that software meets the needs of the business:

  • What the software does
  • How it does it

Or to use more formal terminology:

  • Functional requirements
  • Non-functional requirements

The functional requirements are the business behaviors that provide value to the customer. So for example, if we are creating software for a library, we need the ability to check out books, check in books, and keep a library catalog. Each of these has several behaviors behind it, and there are probably other behaviors I am missing.

The non-functional requirements are also known as the “-ilities” that describe things about the software besides its function. For example, testability is how easy the software is to test. Maintainability is how easy the software is to maintain. There are many others:

  • Evolvability
  • Scalability
  • Security
  • Extensibility
  • Feasibility
  • Reliability
  • Deployability

Besides the above, non-functional requirements might relate to the skill level of developers required to maintain the system, or the structure of the development teams that maintain it. (e.g. the presence of an off-shore development team)

The central claim of software architecture is that you can implement functional requirements in any architecture, but different architectural styles have different non-functional characteristics. What do I mean by that? They provide software structures that emphasize some characteristics at the expense of others. This is a classic engineering trade-off. The job of the software architect is to select an appropriate software structure that matches the non-functional requirements, which should also match the business expectations.

My term “software structure” would overlap with the term architectural pattern, which I first heard about from Mark Richards. Some examples of architectural patterns include:

  • Layered architecture
  • Microkernel architecture
  • Pipes and filters architecture
  • Service oriented architecture
  • Event driven architecture
  • Space-based architecture
  • Microservices

Ah yes, microservices, the buzzword of the last few years. I don’t have time to go into microservices in this blog post, but check out Jimmy Bogard’s Microservices FAQ for more info.

I mentioned earlier that there are different aspects of providing structure. In addition to architectural patterns, which affect the code (technology), there are:

  • organizational patterns (e.g. are teams organized by business capability or by similar job roles? What is the mapping between products to teams?)
  • practices (such as TDD or Extreme programming)
  • culture (probably another blog post altogether)

Ideally all four aspects of structure work together to accomplish the desired outcome. (I’m not sure of the origin of this list of 4 items came from, if anybody knows, leave a note in the comments.)

Code generation

Code generation is a common tool in software. Usually it sits behind the scenes, just a layer of automation translating from one programming language to another. But sometimes it becomes useful to build your own code generator. In a project at work, we’ve implemented a code generator based off a DSL (domain specific language) that generates an implementation of our domain contract. There’s a lot of meaty terms in that last sentence, let’s spend some time and break them down.

Common use case: compilers

First, I said that code generation is common, but behind the scenes. That’s because most compilers have some form of code generation happening in them. So for example, compiling Typescript for a Web app, or compiling Swift for an iOS app.

Let’s talk a little more about compilers. A compiler’s job is to take source code that humans can read and produce output that a computer can run. This is broken up into parts:

  1. Parsing the source code
  2. Stuff
  3. Generate output.

Parsing the source code is all about understanding text and turning it into a format a computer can do stuff with.

Stuff could be interesting things, like in Typescript, adding static type checking to vanilla Javascript. This usually involves manipulating an abstract syntax tree. For example, you could enforce rules in the tree and make sure everything is valid.

Generate output is the code generation part. The tree needs to turn back into something that the computer can run. For Typescript this is generating Javascript, for Swift, this is producing machine code.

A quick example

2 + 2

Given the above syntax, parsing turns it into a syntax tree:

Then for stuff maybe we want to enforce that a “Plus” always has “integers” connected to it. In this case, our tree looks good.

For generate output, we need a rule for what to do with a “Plus”. I’m making up this rule:

  • Load the left hand number into the computer.
  • Load the right hand number into the computer.
  • Add them.

So we apply that rule to our tree and get:

  • Load a 2 into the computer.
  • Load a 2 into the computer.
  • Add them.

Which is approximately what machine language looks like 🙂 I linked to NAND to Tetris which my friend Devin Mork showed me. It reminds me of my computer architecture course in college.

Domain Specific Languages

A domain specific language is making your own programming language. This lets you (1) create a language that is more meaningful and specific, and (2) produce generated code that does something useful.

One example is defining an API between two microservices. You could exchange URL’s back and forth between developers:

https://example.com/api/add-numbers
https://example.com/api/get-result

But then you’d also need to agree on an API body format (maybe JSON), whether things are a GET or a POST, and a myriad of other details.

What if instead you could use a DSL that let you define the following:

api Math {
    command add-numbers(
        leftHand: int,
        rightHand: int
    );
    query get-result(
    );
}

This defines something meaningful to the domain (we have an api with a command and a query), but hides the implementation.

The implementation could be a Web API, it could be a “REST API,” it could be something like Apple’s XPC. This implementation would be built out as a set of code generators, but it is not part of the domain definition. So for example, we could generate a Jersey controller:

@Path("/Math")
public class MathController {

    @GET
    @Path("/get-result")
    public Response getResult() {
        //something
    }

    @POST
    @Path("/add-numbers")
    public Response addNumbers(@QueryParam("leftHand") String leftHand, @QueryParam("rightHand") String rightHand) {
        //something
    }
}

This is a partial example, you’d need a bridge between the generated code and hand-written code. For example, maybe “something” is a call to an interface that a developer will implement by hand.

This is not a new idea, Martin Fowler wrote a book “Domain Specific Languages.” For web APIs see also grpc and api blueprint. If those meet your needs, I would use those instead of making the investment in building your own DSL and code generator.

Event sourcing and CQRS

As part of the #2018TechReadingChallenge, I’ve been working my way through Microservices Patterns, by Chris Richardson (currently a Manning MEAP).

Event sourcing and CQRS are two key architecture concepts in the book that have less to do with the “micro” in “microservices,” and more to do with software architecture in general. While they are often mentioned together, they are separate concepts, and solve slightly different problems.

Event sourcing fits in the general category of an append only model, which is a way of persisting the data in your application. Instead of storing the current state, it stores history that led up to the present. It stores these using events, which are immutable, and represent business intent. Events record a thing that has happened.

A related concept in domain driven design is an aggregate, which rebuilds state by querying the event history.

CQRS stands for Command Query Responsibility Segregation. This relates to an OO principle, command query separation, which classifies methods into “commands,” which mutate the data (but does not return data), and “queries,” which return data only. In CQRS, we extend this principle to the design of a subdomain, and separate the responsibility of writing the data from reading the data. The two sides often communicate using events.

It solves a “stale data” problem in a collaborative domain. A collaborative domain is a business domain where multiple users are working together on a set of data, and expect that data to be coherent. The data is only as stale as the lag between the command side and the query side. However, by separating these sides and putting a visible part of the architecture between them, we have raised awareness that there will always be stale data in a collaborative system.

Another advantage to CQRS is that the read side and the command side can be scaled independently, so for example, if the number of queries is vastly greater than the commands, you can add more nodes that handle the queries.

For more on these concepts, see the CQRS Journey page on Microsoft, specifically the Exploring CQRS and Event Sourcing e-book. I’m also drawing from Patterns for Building Distributed Systems for The Enterprise, by Michael Perry on Pluralsight.

2018 Tech Reading Challenge

Many of us are challenging ourselves to grow in the new year. One area of growth I strive for is in my career as a software developer. To that end, I’m sharing my 2018 reading challenge. I hope you will find it stretches your technical ability.

How it works

  • Choose a reading goal early in the year and set your pace accordingly.
    • The beginner reading challenge is 1 book a quarter.
    • The advanced reading challenge is 1 book a month with 2 months off.
  • Choose the books and read them in any order, checking them off as you complete them.
  • Post about your progress on Twitter using the hashtag #2018TechReadingChallenge. (Cross-post on Slack too)

Ideas

  • Define your definition of “read,” since some technical books are pretty dry if you try to read them from cover to cover.
  • Ignore the categories and use #2018TechReadingChallenge about any software development book you read this year.

2018 Reading challenge (beginner—4 books)

  • A book about unit testing
  • A book about a framework, tool, or language
  • A book recommended by a coworker
  • A book published in 2018

2018 Reading challenge (advanced—10 books)

  • All the books in the beginner list plus:
  • A book about leadership
  • A book about an abstract concept
  • A book about software architecture
  • A book about agile
  • A book from the 80s
  • A book with an animal on the cover

 

Finding concurrency bugs in Objective C

Introduction to concurrency in Objective C

We can introduce concurrency to a process by creating threads. Within a process, threads use the same memory space and the same instruction set. Each thread has its own position within that instruction set. A thread is in one of 3 main states: running, ready, or blocked. Each thread has its own register state and its own call stack.

In OS X 10.6, Apple introduced Grand Central Dispatch, a thread pool technology that simplifies the use of threads. GCD introduces the concept of queues and tasks. A process can have multiple queues, where each queue has one or more blocks. A block is a task for execution on that queue. From a programming languages perspective, a block is a closure (a function plus a binding environment). As a user of the queue API, you simply dispatch blocks on the queue, and GCD manages the running of queues.

Behind the scenes, GCD will create and destroy threads as needed in order to handle the workload of the queues the user has created.

In the case of a serial queue, one block runs at a time. In the case of a concurrent queue, GCD may concurrently execute blocks placed on that queue on different threads.

There are two ways to place a block on a queue.

  • dispatch_sync places a block on a queue and waits until that block completes
  • dispatch_async places a block on a queue and continues immediately

Finding concurrency bugs in Objective C

In code that uses multiple queues, one common concurrency bug is deadlock. Deadlock can happen when there is contention between two queues.

One example of deadlock is when blocks on two separate queues make dispatch_sync calls to each other.

Let’s say we have two queues, the sending queue and the manager queue. The sending queue has a method called beginTransaction, and the manager queue, which oversees both sending and receiving, has methods for managing crypto keys, including one called updateCryptoKey.

beginTransaction looks like this:

make sure the transaction is valid
dispatch_async(sending queue, ^{
    ensure no other transactions are in progress
    get crypto keys for this transaction
    put the transaction in our data structure
    start the transaction
});

updateCryptoKey looks like this:

make sure key being submitted is valid
dispatch_sync(manager queue, ^{
    is there an existing key?
    if not, add the key
    if yes, remove the existing key and add the new key
});

There is a potential deadlock between the sending queue and the manager queue. Each block has a dependency on the other queue. For the block in beginTransaction, the step of getting the crypto keys will do a dispatch_sync to the manager queue. Meanwhile, for the block in the manager queue, the step of removing the existing keys will do a dispatch_sync to the sending queue as part of cancelling any existing transactions under the old key.

These queue dependencies aren’t immediately obvious, because they are hidden behind function calls.

When finding deadlock between queues, it doesn’t matter how a block got placed on a queue. Here we have one block that was placed using dispatch_async, and one that was placed using dispatch_sync. What matters is what happens once that block is running. Since these two blocks are on separate queues, they can be running simultaneously. Then one of them does a dispatch_sync to the other, and it waits, because that queue is not free at the moment. Then the other does a dispatch_sync back to the first, and it waits too, because that queue isn’t free either. This is a deadlock.