Tuesday, September 22, 2020

Spot the Bug - FOR Loops

The next bug in the Spot the Bug series is one that is seen quite often from new programmers. Can you spot the bug?

This code contains a class that adds two 2x2 matrices together. The class also includes a print method for displaying the matrices in the console. The main program passes in two sample matrices to add together. The bug in this code is that the indices in the AddTwoMatrices method are flipped for the additionalResult object.

This bug is common with new programmers. Unfortunately there is no C# language construct to prevent us from creating this kind of bug. I've included it in this series because it presents a great conversation for a mentor to have with a new developer about programming techniques that can be employed to lessen the chances that a bug like this gets released.

The first thing I would highlight is that this code needs unit tests. Math operations are straightforward to test because they should be operations that have no side effects. The code doing the math operation should also be pure, meaning for the same set of inputs, it always produces the same output. These conditions make unit testing math methods straightforward and these methods should always have accompanying unit tests.

The other thing I would highlight here is that better variable naming may make it easier to see the bug when it does get introduced. (Generally speaking, I don't mind simple counter variables like these in FOR loops being single-letter variables.) More descriptive variable names for the loop counters will make it easier to read the code and reason about the code, making it easier for a developer to see a problem as it is being introduced.

Here is the code with the bug corrected and more descriptive variables to make it more readable.

Even though C# doesn't provide mechanisms to prevent this type of bug, there are some good programming techniques that can be taught and discussed when bugs like this arise.

Friday, September 11, 2020

Spot the Bug - File Globbing

The bug in this post comes from the world of scripting. Powershell is used for the code below, but this bug could be introduced in any scripting language that is commonly used for task automation. Can you spot the bug?

The code above is a fairly straightforward script. It gets a list of files from fileDir, iterates over each file moving it to two different locations. After looping through the files, it removes the files from fileDir.

The bug in this script is that the removal of the files is done using a wildcard to locate the files to delete. If the number and size of the files being moved is large, the iteration over each file will take some time. New files can be placed in fileDir and those files will be deleted by the final script line using the wildcard. If that scenario happens, data can be lost!

In the code above, the bug is fixed by deleting files by file name as a part of iterating over the list. This approach ensures that you only affect files discovered in the first line of the script, and newly arriving files can't be accidentally deleted.

This example had to be done in Powershell because the C# libraries for working with files and directories don't allow the use of wildcards, therefore preventing a developer from creating this bug. This is another example of the language construct preventing developers from introducing bugs.

Friday, September 4, 2020

Spot the Bug - Using Statement Part 2

I introduced the "Spot the Bug" exercise previously. This is the next bug in the series.

Below is a class named DataWriter. This class is responsible for opening a file, closing a file, and writing data to a file. Below that class is a console program that exercises the DataWriter class. Can you spot the bug?

In the console program, I forgot to call the method to close the file. This is another example of the using statement in C# protecting us from creating code like this example. In the previous example, we saw a similar bug where it is very easy for the developer to forget to close a connection or file handle, using up precious resources. The using statement isn't a perfect solution, but it does push the developer to keep responsibilities grouped appropriately and it ensures that the garbage collector can do some amount of cleanup in the system when resources are no longer needed.

Below is code that represents a better solution.

The code above uses the using statement to handle the opening of the file. The developer is still responsible for closing the file, but the using statement will make sure that the StreamWriter object is disposed.

The other thing to notice in the better code example is that the DataWriter class was completely eliminated. Because the DataWriter class split up the file writing responsibilities too granularly, once those responsibilities were merged back together, the class was no longer necessary. This example highlights the fact that sometimes our designs aren't quite right, and we shouldn't be afraid to remove classes if they aren't necessary.

Monday, August 31, 2020

Accessing Config Data in Azure Functions with F#

I've been building an application using Azure Functions with F# as the API backend. The Azure Functions access data in Azure Storage to return to a single page web application. For development purposes, I've had the connection string hard-coded in the F# files that needed it. I'm at a point where I need to start deploying this app in a production manner, so I need to get the connection strings out of my code. The documentation for Azure Functions recommends using environment variables in production, and the local.settings.json file for development. Reading these values should have been trivial, but wasn't. This post documents how I was finally able to read data from environment variables.

Most of the documentation that I was able to find online pointed to using the Environment.GetEnvironmentVariable("myData") syntax. I was absolutely unable to get this call to work for me in my F# function. Fortunately I'm a member of the F# Slack community. I reached out for help on this issue, and Zaid Ajaj gave me a few things to try. This experimentation led to the solution below:

In the function above, I create a new ConfigurationBuilder instance and configure it for my app. To properly configure it, I needed to include another parameter for the function call, (context: ExecutionContext). This parameter gives me access to the runtime directory with the property context.FunctionAppDirectory. This allows me to then inject the local.settings.json file. (Note the boolean value true in the call to AddJsonFile. This true indicates that this file is optional, which is necessary in a production setting when this file will not exist.). Once I build the ConfigurationBuilder, I'm able to access local settings and environment variables using the array accessor syntax, config.["BoogyMan"].

I'm not sure if this is the "approved" Microsoft way or the best way to read environment values in an F# Azure Function, but it does work. If there is a better way to to this, I would love to see it.

Friday, August 28, 2020

Spot the Bug - Using Statement Part 1

Several months ago, I was reviewing some C# code of one of my teammates when I realized there was a subtle bug in the program. The bug was not easy to see right away. I realized that C# had certain constructs in place to make it difficult to create bugs like this. These constructs are great, but they mask certain design principles and make it possible to learn programming without learning the underlying design principles. I reviewed the bug in question with my team during a code review and made it kind of a game to find the bug and talk about the underlying design principles.

I did a talk titled "Spot the Bug" for the Code PaLOUsa conference last week. The talk was geared for two primary audiences. The first audience is developers that are earlier in their career. This audience benefited from this talk in seeing common coding mistakes and learning some common design principles. The other audience for this talk was developers later in their career or developers in positions to mentor others. This audience was shown an approach for highlighting some of these common mistakes and seeing ways to address them with their creators.

This is the first topic that I covered in the talk.

Below is a class to access a database. It has methods to open and close the database connection as well as a method to work with the data. Can you spot the bug?

Most C# developers are looking at the code above and thinking to themselves, "Why didn't he do this with a using statement?" These developers are absolutely right. The bug in the code above is that the DataGatherer class separates the responsibility of managing the database connection between two different methods, effectively forcing the developer that uses this class to be responsible for doing things correctly.

The using block in C# forces a developer to keep the responsibility of opening and closing database connections in the same code block, mostly ensuring that the right things are done in regards to the database connection. (It is still possible for a developer to forget to close the database connection within a using block, but the using block will force the connection closed and dispose of the connection. It just may take the garbage collector longer to get to it.)

The better code is below:

You'll notice in the better code that there is a lot less code! Keeping the opening and closing responsibilities together inside the using block, reduces the amount of code in the class and creates a much better program.

This subtle bug is easy to overlook if you learned to work with a database after the creation of the using construct. The using construct is a way to talk to new developers about class and method responsibilities and the need to keep similarly oriented responsibilities together.

Tuesday, August 11, 2020

Moving to Functional Thinking

I've been working on an educational-based site to help teachers evaluate students. I'm using F# for the backend API, and the regular use of a functional programming language has me thinking differently about how I code.

I recently needed to test a value to see if it is a GUID. If the value is a GUID, I want to use it. If the value isn't a valid GUID, I want to create a new GUID. In C#, this would be done like this:

if(Guid.TryParse(maybeGuidValue, out validGuidValue)
{
 myObject.ID = validGuidValue;
}
else
{
 myObject.ID = Guid.NewGuid();
}

This code is fairly straightforward, but a little problematic. First, the TryParse() method takes a string to test, maybeGuidValue, and an out parameter. If the method succeeds, meaning, the value is a valid GUID, the method returns true and the value validGuidValue is populated with the value in a GUID data type. If the value passed in is not a GUID, the method returns false and the out parameter should be ignored.

This code is tried and true in the C# world, but it isn't great. First, I'm not a fan of out parameters. This code construct is not pure or immutable since the method is returning one value as part of the method call and mutating another value by sending back an out parameter. Code like this requires special attention and is difficult to wrap in good unit tests.

In contrast, F# handles the same concept in a more intuitive manner. The code to TryParse a GUID looks like this:

let didParse, validGuidValue = Guid.TryParse maybeGuidValue
let ID = if didParse then validGuidValue else Guid.NewGuid()

This code calls the same TryParse method that is used in the C# code (F# shares the .NET libraries). Instead of having a single return value and an out parameter, this function returns a tuple with the boolean result of the parse operation along with the possibly populated GUID value. This construct is easier to see and much easier to unit test! The other obvious difference in the F# code is that it is only two lines! One line parses the value and the other line makes the assignment.

The F# code seems easier and much more intuitive, but it requires a change in thinking. The first change in thinking is embracing the power of tuples. Since tuples are relatively new in C#, it takes a mental effort to remember they are critical to F# and truly powerful. The second change in thinking is that if statements are expressions in F# as opposed to blocks in C#. An expression results in a value, allowing the assignment of ID, no matter how it is generated, to be made in a single line of code. The block construct in C# allows for branching, but the assignment code needs to be implemented in each logic branch. The block approach results in more code that can lead to ambiguity in how a value is assigned.

As I get more practice using F#, I am really enjoying the simplicity of code and the power the language has for me.

Tuesday, July 7, 2020

F# Kata - Merging Two Lists

I recently did a C# kata where I had to write a function that merged two arrays of strings together, ensuring that the resulting array only contained distinct strings. It was straightforward to write in C#, but I kept thinking I could do it easier in F#. I was right. My code is below:

Wednesday, June 17, 2020

Cloud-Native Design Techniques - Scale

Designing systems for the cloud, or to be cloud-native, is all the rage right now. There are numerous articles written about this, but most of them come across like partial cooking instructions where the reader is expected to know the recipe already. I want to talk about cloud-native design, but not make any assumptions about prior knowledge.

For the past 14 years, I've worked on a system that processes electronic payment transactions. When we built this system, we specifically designed the applications to handle scale. We were betting the business on being able to scale the software linearly with business growth. What exactly does this mean and how is it done?

First, let's define what it means for an application to "scale." In the simplest terms, an application is able to "scale" when it can handle more requests or load without changing the application itself, without failing, and generally without significant changes to the infrastructure that executes it. Let's take for example an order handling system. The code below loosely defines this system:

We can see in the code that three activities occur for a new order: 1) the order is validated, 2) the order is billed, and 3) the order is shipped. In this example, the order validation must occur for the other two activities to happen, but not all of the activities are chained together. We can see that the process of billing an order interacts with a database. Database interactions generally require requests over a network and can be constrained by the resources available to the database server. This operation could be slow or contentious. If orders occur infrequently, this approach is fine and will work well enough. If orders are streaming in rapidly, this approach will either cause significant delays in processing orders, or it will crash the entire system as resources get overwhelmed.

There are three ways that we could easily modify this system to handle scale. The first approach would be to upgrade the server it runs on, or to run it on multiple servers. A hardware-based approach is the easiest for developers because they don't really have to do anything, but it may not be the best long-term solution. Upgrading the existing server may allow for more orders to be handled for a while, but the system will fail again as the number of orders increases. Running this system on multiple servers may work, or it may cause more contention in shared resources, like a database, that causes the system to fail even sooner. If this application is running on a virtual machine in a public cloud, modifying the underlying hardware is an expensive approach to making it scale.

A second approach would be to run the HandleOrder method in a new thread for each order. This approach isn't terrible, but it isn't as easy as it sounds. Simply creating a new thread for each order may work, but that assumes that everything in our HandleOrder method is thread-safe. Most database operations are not thread-safe, so the order billing operation will be an issue. In my experience, a lot of code is only thread-safe by accident, not because it was intentionally written that way. Using threading to scale this application may be a valid approach, but it requires diligence and understanding to get it right. The other limitation to this approach is that at some point, the resources of the server will be exhausted and the app will no longer scale. When this happens, a hardware approach will be necessary.

The third approach to making this system scale would be to break up the operations and handle them independently. We see that the order validation is the only required step for billing or shipping an order. The system can be broken up into three pieces, two of which can be called independently. The code below illustrates how the separation could occur:

The code above separates the billing and shipping calls and allows them to run simultaneously since they aren't dependent on each other. (This code is overly simplistic to illustrate my point.) Breaking the system into constituent parts allows us to scale the BillOrder function separately from the ShipOrder function. As our load of orders grows, we can run the BillOrder function separately from the ShipOrder function, using the HandleOrder function as an orchestrator for the work. In a public cloud environment, this approach would allow us to run the billing and shipping functions as either serverless functions or independent Docker containers. (Either serverless functions or independent containers are cheaper options than running full virtual machines in public cloud environments.)

Designing for scale requires an architect to see the divisions in an application and to separate the system along these divisions. Once that separation is properly done, the system is better prepared to take advantage of cloud techologies to handle scale in a more cost-effective manner.

Wednesday, June 3, 2020

F# and Azure Functions

Lately I've been creating tools for my wife to use with her classroom to help with distance learning. I've implemented these tools using Azure Functions. Azure Functions are the serverless computing option on the Azure platform. I picked these because I'm familiar with Azure and they are a quick, cheap option for standing up an API.

Another reason I picked Azure Functions is because they support creating functions in F#. I have been able to create a function project and function in F#, and at times run it locally, but I have struggled to get an F# function working in the cloud. I assumed that this issue was a combination of spotty support for F# in some areas, and my lack of understanding about how to get F# to play nicely as an Azure Function.

I have a new project I want to start as a series of Azure Functions, so I wanted to give F# another try. In my reading, I found this step-by-step guide by Luis Quintanilla for manually creating Azure Functions in F#. Luis does a great job walking the reader through creating a function in F#, without relying on the built-in templates provided by Microsoft. I found this guide fantastic because it gave me a better understanding of the composition of an Azure Function, AND it got me closer to successfully deploying an F# function.

The guide isn't perfect. It's missing some steps around adding necessary packages to the project that are referenced in the function itself (easy to do). This "gap" helped me discover some version discrepancies between by installed version of the .NET Core SDK and the Azure Function CLI tool. At first it appeared to be an issue with my project templates being out of date, but updating the templates didn't seem to fix anything. Ultimately, I updated my version of the .NET Core SDK to the latest version, and I upgraded my version of the Azure Function CLI.

At this point, my function that I created by following the guide worked locally and in the cloud! I went back and re-created my function using the command-line tools, and that worked as well. Using version 3.1.300 of the .NET Core SDK and version 3.0.2534 of the Azure CLI tools (which are targetting the Azure Function Runtime version 3.0.133353.0), the following steps will successfully create an Azure Function in F# that will work in the cloud:

mkdir HelloWorld
cd HelloWorld
dotnet new func --language F#
dotnet new http --language F# --name PrintHello
dotnet build
func start

The logging displayed in the console from the func start command will provide the URL for the local function. Pointing your browser to that URL will activate the locally running function.

I'm now ready to start creating larger applications in F#!

Tuesday, May 26, 2020

F# Records

My previous post F# - Refactoring to Functional - Part 1 garnered some attention for my use of the struct data type.

Tobias Burger pointed me at the correct implementation of F# records for my data passing. He suggested using ordinary records as my data type instead of structs since they are a more idiomatic approach to F#. It's taken me a little while to get back to the code, but I modified my data types to be records. The code is below:

I have to say "Thank you" to Tobias. His criticism helped me better understand F# records, and it made my code shorter and cleaner!

Friday, May 22, 2020

Microsoft's Project Tye

I just read through a blog post about Microsoft's Project Tye. This is an experimental project that helps developers run multiple apps under a single command. It also has some service discovery elements and a deployment capability for Kubernetes.

I'm putting this link here so that I can find it again later. The ability to coordinate the running of multiple projects (either in a microservices world or just with a frontend and backend components) is compelling enough. The service discovery and Kubernetes deployment capabilities make this really worth understanding.

Wednesday, May 13, 2020

F# - Refactoring to a Functional Design - Part 1

Shortly after the coronavirus pandemic started, I started trying to create web-based tools for my wife (who is a preschool teacher) to evaluate the progress her students are making with distance learning. I reached for Azure Functions first since they are quick for implementing a publicly visible API. As I created and worked with this new API, I needed a way to test the end points to make sure they worked the way I needed them to work. I decided to create a tool in F# to do this testing.

(I know tools like Postman exist and are great. As I was working on the application in question, I was working on a brand new laptop, and I didn't want to go nuts installing software. Also, as much as I like Postman, it has a few drawbacks. The most notable two for me are the fact that changes to tests don't appear in a text file until the user exports the tests. This disconnect results in me forgetting to update tests from Postman with my source code. The other drawback is that additional third-party tools are needed to run Postman tests from some popular CI/CD tools. I recently ran into an issue with one of these tools where the wrapper for the CI/CD pipeline was incompatible with an NPM module needed to run Postman tests. This incompatibility forced me to remove the tests from my build pipeline. That shouldn't happen.)

My first pass at the code in F# is like what I have below: This code is alright, but you can see that there is a lot of duplication for each endpoint to call. This code is also more difficult to read and maintain. I would like an easy way to define an API endpoint to exercise, and just have it tested.

The first refactoring I will do is to remove the duplication of the HTTP call. In the code above, the HTTP calls are not terribly onerous. These calls are a single line of code for each endpoint, and that's not bad. What I don't like about this code is that the HTTP dependencies are mixed with the rest of my tests, and I have to include the HTTP method calls with my definition of the end point to test.

The first step for me was to create an abstraction for the endpoint itself. I want to use an F# type for the endpoint, but I don't want this type to be a class or to contain any logic. To achieve this purpose, I used the F# struct type. This element allows me to define value fields in my type, along with a constructor. My Endpoint struct is below: This "type" gives me a Name so that I can identify my test externally, a URL to hit, the HTTP verb to use in the call, and a "body" for sending in data when necessary. (For this post, I won't be using the Body property.)

Now that I have an endpoint abstraction in place, I need a method to make my HTTP call. Since this is a single line of code in my original file, it should be a pretty short and straightforward method. The method looks like:

let GetEndpoint (endPoint: Endpoint) =
    Http.Request(endPoint.Url, httpMethod = endPoint.Verb)
This method takes in an endpoint. I wanted to ensure that this method stuck to my endpoint abstraction, so I specified a type with the ": type" syntax. Since every line of code in F# is an expression, this method returns the result of the Http.Request method, which is an HttpResponse object.

With my endpoint abstraction and my function to make HTTP calls, I can modify my original testing code to look like this:

This code doesn't look a lot better than the original code, but it is a little easier to see the endpoints that are being tested. This code also shows me that I still have a good bit of "boilerplate" stuff with my testing of the response data. The testing code will be the subject of my next post.

Wednesday, May 6, 2020

Quick Intro to Azure Functions

I've been working on a small web application for my wife to help her evaluate her preschool students. I created this application as a single page application (SPA) and backing APIs. For the SPA, I used VueJS. For the backing API, I used Azure Functions.

Azure Functions are Microsoft's serverless computing offering. They allow a developer to create functionality based on several triggers. Triggers can be HTTP, queues, Azure Storage, and so on. One of the most commonly used triggers is HTTP. A function based on an HTTP trigger fires for a web request. I think about this like a Model View Controller (MVC) application with the models and controllers and none of the boilerplate code that goes with a full API application.

An Azure Function application consists of a minimum set of files. Those files are:

host.json
local.settings.json
FunctionApp.csproj
Function.cs
Where "FunctionApp" and "Function" are the names you give your Azure Function.

The host.json file contains runtime information, including the version of Azure Functions being used and things like logging configurations. The local.settings.json file contains application-specific settings including connection strings. The CSPROJ file is a fairly typical C# project file that specifies any package dependencies and things like that. The CS file is the "function" itself. It looks like a typcial static C# class except for an annotation and a specific method signature.

The "function" that serves as a trigger for your function app is annotated with [FunctionName("MyFunction")]. This annotation specifies the publicly visible name that your function will have. The function signature is specific to the type of trigger for your function. Since I used an HTTP trigger, my function signature looked like:

public static async Task Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req, ILogger log)
This signature tells the runtime that I am using an HTTP trigger that responds to HTTP GET and POST requests. I don't specify a specific route. The HttpRequest and a default logger are supplied using the dependency injection that is baked into the service.

When you generate your first function, a default "Hello World" app is created. This application illustrates how to work with the HttpRequest object passed into the function. It also shows how to create the IActionResult to pass output from the function.

For my specific Azure Function app, I put just enough code into the function itself to handle the request and return a response. I created service and repository classes for handling the business logic and data movement for the calls. This approach worked very well for me!

This approach to API development allowed me to get something created and deployed very quickly. I was able to utilize some basic tooling from Microsoft to build, test, and deploy my API very quickly.

Monday, April 20, 2020

F# Active Patterns

My company deals with file parsing quite a bit, and I've been playing around with F# to see if I could write a better file parsing library using it instead of what we currently use in the C# world. This experimentation led me to F# Active Patterns.

F#, along with other languages, has a concept of Pattern Matching. Pattern matching allows a programmer to transform data by matching patterns in the shape of the data automatically, without a lot of if-then-else branching logic.

Pattern matching, using the match keyword, functions a lot like a switch statement in C#. A typical pattern matching statement in F# would look something like:

let filter1 x =
    match x
    | 1 -> printfn "The value is 1."
    | _ -> printfn "The value is not 1."

filter1 5.   // The value is not 1.
filter1 1.   // The value is 1.
This code defines a function, filter1, that takes an integer. It checks to see if the integer is a constant value of 1. If it is, it prints a message telling us it is 1. If it isn't, it tells us the value is not 1. This code illustrates a constant pattern, and their are many additional options. The other options are described here.

The example above, and the other basic pattern matching, is great, but not enough for some of my file parsing needs. Many of the files that we need to parse are fixed-width ASCII text files. Parsing the data requires defining fields in a particular order with a specified size. This aspect is unavoidable, but I wanted to find an elegant, way to approach this problem.

This is where I found Active Patterns. Active Patterns allow you to define a named pattern and apply it in a match statement. This pattern will match and parse the data if defined correctly. It is also the mechanism for matching with regular expressions, which lend themselves well to my file parsing problem. An Active Pattern looks like this:

let (|EmailMatchActivePattern|_|) input =
    let m = Regex.match(input, "(.*)@(.*)")
    if m.Success then Some m.Groups.[2].Value else None
This construct creates an Active Pattern named EmailMatchActivePattern and it takes in input to work against. Inside the definition, I'm using a regex for a basic email pattern (.*)@(.*). If the regex matches successfully, I return the second grouping. (The indexing starts at 0, but the 0 index contains the entire string that was tested with the regex. The following indices contain the matched classes from the regex.)

Now that I have a named pattern defined, I can use it in a normal pattern matching scenario to match a line from a file and parse the data in that line. I do that with code like:

let parseLine line =
    match line with
    | EmailMatchActivePattern (domainName) -> printfn "This is the domain in the email %s" domainName
    | _ -> printfn "Not an email addres."
This code parses a line of text to find an email address. If the line matches, it parses the domain portion of the email address into domainName. I can then reference domainName in the subsequent function call (a print statement). This code matches and parses the data at the same time.

I took this concept further to explore the file parsing that I described at the beginning. In this exploration, I read lines from a file, match the first two characters of the line to determine a record type, and then parse the line into tuples with individual values. This construct allowed me to define my file format as a couple of predefined regular expressions and to match the records and parse them at the same time. The source code for this is below.

Active Patterns is an F# construct that can be used to dynamically match and parse data. This is a powerful construct that can be used to build tools to parse files and other data structures with a minimal amount of code.

Monday, April 13, 2020

F# Sequence

When I was learning how to work with files, I was introduced to the seq keyword. This keyword is used to create a Sequence in F#. Sequences are powerful because they basically implement the IEnumerable<T> data structure in .NET.

IEnumerable<T> in .NET is an interface that exposes an enumerator of type T. This means that any data structure that implements IEnumerable<T> can be used in a loop to process the contents of the data structure. This construct is a powerful tool used for processing collections of things in the .NET space. Sequences are the manifestation of IEnumerable<T> in F#. Since Sequences are the generic implementation for collection data structures in F#, I need to understand them better.

When I was introduced to Sequences when working with files, I didn't fully understand how to iterate over the list and get direct access to the items in the collection. I was using the Seq.filter function to "iterate" over the list. Seq.filter will pass over each item in the Sequence, but it creates a new Sequence containing the items filtered by the function that is passed into it. I had incorrectly assumed it would output the filtered item. It does not. The filter function creates a new Sequence!

One approach that would have worked for my purposes was using a for loop. This construct loops over every item in a collection and executes the code in the containing block for every item in the collection. for loops work very well, but they generally aren't considered a good approach for a functional programming mindset (see the note below).

Many functional programming languages have a map function (or an equivalent). This function executes the code passed into it as a function on each element in the collection. F# has the Seq.iter function to do this iteration. This function iterates over each element in the Sequence, applying the function that was passed in to the element. This is what I was missing!

Now that I can simply iterate over a collection, I can easily read a file and print each line to the console. The following code does it:

open System
open System.IO

let readLinesFromFile fileName =
    seq { use reader = File.OpenText fileName
        while not reader.EndOfStream
            do yield reader.ReadLine() }
                
[<EntryPoint>]
let main argv =
    let lines = readLinesFromFile "test.txt"

    lines
        |> Seq.iter (fun x -> printfn "%s" x)
        
    0 // return an integer exit code

I feel better about my understanding of Sequences and how to use them in basic scenarios. It isn't everything I need to know, but I feel better equipped to use F#.

NOTE: Many functional programming languages don't implement looping constructs in the language definition. Those languages accomplish the same thing with a "map" function to iterate over a collection and sophisticated function passing to process the contents of the collection. Recursion is also common to avoid looping constructs. These approaches are considered more efficient in functional programming.

Friday, April 10, 2020

Learning F# - Odds and Ends

Now that I've gotten through my basic steps to "learn" a new programming language, I'm ready to actually start learning the language. What do I mean by this? When I studied martial arts, my teacher always said "You don't start learning something until you've done it 10,000 times." This seems like an exaggeration, but it's true!

When learning something, one needs time with it to truly learn it. One learning to play an instrument must practice regularly to get good at playing the instrument. Someone learning to cook has to cook regularly to get better at it. It is the same with learning a new programming language. You need to work with it regularly to get better at it.

Now, repetition is good, but that repetition needs feedback for improvement to happen. For me with programming, unit testing provides that feedback. Unit tests help me know if my solution works, if my code is too complicated, and if my design is moderately reasonable. This is why learning a unit testing framework in a new language is an early step for me.

The rest of the "learning" steps that I've documented are a means to an end. Knowing how to interact with files, or a database, or a web API in a new programming language provides me the tools to use the language in real-world scenarios.

The .NET ecosystem provides some easy, low-cost ways to use F# in real-world scenarios. First, the .NET command line tool provides a REPL (Read-Eval-Print-Loop) for F#. This tool allows you to run F# code interactively in a terminal window. You access the REPL with the command:

dotnet fsi
You can access packages using the #r directive. In the REPL environment, the #r directive references a DLL somewhere on disk, using the full path to the DLL. You can then use the Open keyword to make the namespace available from the referenced DLL.

The REPL also provides an ability to run F# scripts. An F# script file is designated with an fsx extension. It can be run from the command line with a call like:

dotnet fsi my_fsharp_script.fsx
This can be handy for day-to-day scripting tasks, and provides a way to practice with F#. (I documented a recent instance of this for me here.

As I continue to work with F# and really learn the language, I will continue to document what I'm learning.

Tuesday, April 7, 2020

Learning F# - Step 6

The next step in my learning journey for a new language is to figure out how to access web APIs.

When I searched for how to access web APIs, I was brought back to FSharp.Data. This library also provides interfaces to make HTTP calls! (I was a little leery about using this package after my experience with the data access!)

As it turns out, this library works really well for HTTP calls! Once I added it my project with the call:

dotnet add package FSharp.Data
and referenced the package in my code with:
open FSharp.Data
I was able to start accessing web sites.

Below is the most basic web request that you can make. In my case, I just pulled up Google and printed the HTML response to the console:

let google = Http.RequestString("http://www.google.com")
printfn "%s" google
This code calls "www.google.com" and prints the result to the console. This is a great start, but I need to be able to do more.

The next step for me is to figure out how to do more complex calls. That was fairly easy too. The following code calls Google with a specific search term and prints the result to the console:

let searchResults = Http.RequestString("http://www.google.com/search",
                                       httpMethod = "GET",
                                       query = [ "q", "butterflies"])
printfn "%s" searchResults
This code illustrates a few things that we need to know to access web APIs. First, it gives us a way to specify the HTTP verb: httpMethod = "GET". It also shows us how to send query parameters using query. The query parameter takes an array of items. If we needed to emulate a form submission, we can change the HTTP verb to POST and send in a body element with an array of values, just like the query parameter.

Now that I have learned some basic tasks to accomplish in F#, I can start using it to solve some real-world problems to learn the actual language. In my next post, I'll talk about some odds-and-ends I've found along this journey, as well as some good sites for diving into F# and getting a handle on functional programming.

Saturday, April 4, 2020

Learning F# - Step 5

The next step for me in learning a new language is interacting with a database. This step has kicked my butt!

Since F# is a Microsoft-sponsored language, I expected to find a straightforward approach to interacting with a SQL database. That wasn't the case!

When I searched for data access within F#, I was taken to this page. I wanted to work with a basic SQL database, so I clicked the link for SQL Data Access. I was pleasantly surprised to find several options for SQL data access. The first one I chose was FSharp.Data.SqlClient. I chose this one because it looked very similar to the SQL data access provided in C# by the System.Data.SqlClient namespace.

I jumped straight into implementing the sample code on the front page of FSharp.Data.SqlClient. As I built out the example and started to run it, I ran into errors that didn't make sense to me. As I read about this package, I realized it worked with System.Data.SqlClient, and I needed to add it to my program. I did that. I still had errors. As I read more about the errors I was seeing, I found that there is a dependency on mono for this package. In most cases, this wouldn't be a big deal, but it is for me at the moment.

I am doing most of my exploration of F# using .NET Core on a Macbook. As .NET has moved to support cross-platform development, they have relied on mono at times to supply functionality, so it isn't a surprise to learn that mono may be required for some packages still. For this particular learning step in my F# journey, I'm trying to minimize the complications and differences between F# on a Windows platform and cross-platform F#. To that end, I decided to use System.Data.SqlClient for this step.

The first thing I need to do is to set up a SQL database. For this purpose, I'm using SQL Server 2019 (actually running in a Docker image). I need to create a database and a table. Below are the T-SQL commands to do both:

create database FSharp

create table PlayingCard (
  ID int,
  CardValue varchar(5),
  Suit varchar(8)
)
Now that I have a database available, I can start working with it.

When I'm learning to use a new language and interact with a database, I try to do 2 basic operations: insert data and read data. Since I'm using System.Data.SqlClient and I'm familiar with this package in C#, I know that I need to create a SqlConnection, create a SqlCommand with my SQL statement, and then execute the command to perform the operation.

In F#, I start by making sure the package is available to my code with the open System.Data.SqlClient call. I then created my connection string for the database. I encountered the attribute Literal for the first time in F#. This annotation effectively creates a "constant" value. Now I'm ready to work with the database.

C# has a construct called using that allows a developer to open a resource (such as a file handle or database connection) in such a way that the compiler is told to dispose of the resource using the language construct instead of the developer calling it directly. It turns out, F# has the same construct!

I start my database interation by calling using (new SqlConnection(connString)). This instantiates a new SqlConnection. It then passes the connection to the function that I pass into the using call. In my case, I create an anonymous function by declaring it and creating it within the using block. An anonymous function is declared using the fun keyword, followed by any parameters that the function takes.

The first thing I do in my function is open the connection. The last thing I do is close the connection. This ensures that I don't abuse the connections available from the database.

Now, I want to put some data into the database so that I can try to read it back out. I create a new SqlCommand with my INSERT statement (values are hard coded) and the connection. I then execute the command with the ExecuteNonQuery call.

To read the data, I create a new SqlCommand with a SELECT statement to read all rows from my table and the connection. I execute the command with ExecuteReader and receive a SqlDataReader back. To get the values from the SqlDataReader, I iterate over the reader with a while loop and print the results to the console.

I started with an F# console app and added the data access package with the call:

dotnet add package System.Data.SqlClient
Here is the full code:
open System.Data.SqlClient

[<Literal>]
let connString = @"Data Source=.;Initial Catalog=FSharp;User=dbuser;Password=MyStrongPassword"
    
[<EntryPoint>]
let main argv =

    // System.Data.SqlClient
    using (new SqlConnection(connString)) ( fun conn ->
        conn.Open()
                                            
        let insertCmd = new SqlCommand("INSERT INTO PlayingCard (ID, CardValue, Suit) VALUES (3, '2', 'Diamonds')", conn)
        let count = insertCmd.ExecuteNonQuery()
                                            
        let readCmd = new SqlCommand("SELECT * FROM PlayingCard", conn)
        let reader = readCmd.ExecuteReader()
        while reader.Read() do
            printfn "%A %A %A" reader.[0] reader.[1] reader.[2]
            
        conn.Close()
    )
    
    0

If you are familiar with C# and interacting with a database in C#, this code is very comfortable and familiar. It also gives me another tool in my F# arsenal.

NOTE #1: F# has another construct for working with resources call use. It works similarly to using. I have not fully figured out the difference between the two versions.

NOTE #2: My "trick" to open and close the database connections in the same place is a common design pattern when working with resources such as files and databases. This pattern probably warrants its own blog post.

NOTE #3: I don't really like that I resorted to System.Data.SqlClient. It feels like I'm abusing the availability of C# libraries in the .NET space to get through a task. I need to do more research into F# data access to better understand the space and to find a more idiomatic approach to data access.

NOTE #4: Most of the code in this sample is procedural and not functional. I REALLY don't like this fact. I used it because it has the practical aspect that now I can start creating more meaningful applications with F#, but it hasn't helped me get better at thinking functionally.

Wednesday, April 1, 2020

Question about Automated Tests

I received a question from a friend about end-to-end testing. The question was: "Is it wise to try to build end to end test automation around an intermittently unstable web app?" Instead of sending him my answer directly, I thought I would put my answer in a blog post.

In my opinion, I think it is worth having an end-to-end test for the intermittently unstable web app. I don't think it is worth the effort to automate those tests and include them in part of a CI/CD pipeline. Here's why I think this.

I am a HUGE believer in automated testing. I think having an end-to-end test of a system is a good thing. It gives developers and testers an easy way to exercise changes to the application. In some cases, these tests can be used to verify changes to a production system. It is critical to have these tests so that the functionality is documented and can be exercised by anyone with access to the tests.

I don't think it is worth it to automate these tests against an intermittently unstable system. The key words here are "intermittently unstable." If the tests can run repeatedly with immutable results, I would say automate their execution. The trouble with some web apps is that they may take longer to respond in some areas based on time of day or operation performed, causing inconsistent test results due to time outs. It can cost a lot of time and effort to keep this automation running and test results consistent. If this is the case, I don't think the automation effort is worth the time.

I hope this answers my friend's question. If not, I'll keep the thread going here.

Tuesday, March 31, 2020

Learning F# - Step 4 Revisited

I know I should be writing about using F# to connect to a database, but I had to come back to working with files. (Learning the database interaction is taking me longer, and I'll explain why in that post.)

I had an opportunity to use F# at work to essentially "fold" a file at a certain number of characters. (This is a built-in function of *nix OSes, but I was on a Windows box and didn't have my normal set of tooling due to the quarantine.) I needed to read in a file as a string and break it into lines every 170 characters. The mechanism I posted earlier for reading files worked, but it wasn't doing exactly what I needed.

To suit my need, I found the ReadAllText method on the File object in the System.IO package. Below is the code that I put together:

open System.IO

let fileName = fsi.CommandLineArgs.[1]

let fileContents = File.ReadAllText(fileName)

let newString = ""
for i in 1..(fileContents.Length) do
    if i % 170 <> 0 then fileContents.[i-1].ToString()
    else fileContents.[i-1].ToString() + "\n"
    |> printf "%s"
Most of this should look familiar to the code I had in my last post. One big difference is that I actually wrote this as an F# script file (fsx) instead of a straight F# "program." This is why you don't see any actual function definitions in the code. Not being a "real" F# program, the command-line argument handling is different too (fsi.CommandLineArgs). I'll save those items for another post.

The main difference I want to point out is the file operation. In the code above, I read the entire contents of my file into fileContents using the File.ReadAllText method. This change allowed me to work on the contents of the file as a string. That "work" is to iterate over the characters in the file and insert a newline character after every 170 characters.

The fact that I used a FOR loop to do the iteration points to the fact that I have a LOT of learning to do in the functional programming world.

Saturday, March 28, 2020

Learning F# - Step 4

In this next step of learning F#, I want to interact with the file system. Once I can read and write files, I can use a programming language for day-to-day scripting needs, which helps me get more comfortable with the language's features.

Fortunately, F# uses the .NET System.IO library for interacting with files. This fact makes it easy to translate how to work with files in C# to F#. The first thing to do is to "open" the System.IO package in your file:

open System.IO
Now that you have the library available, you can work with files.

Let's start by reading a file. The following code reads a file line by line into a sequence of lines:

let readLinesFromFile fileName =
    seq { use reader = File.OpenText fileName
        while not reader.EndOfStream
            do yield reader.ReadLine() }
This code block reads lines from file fileName and puts them into a sequence. A sequence in F# is a logical series of elements all of one type. Since the last line of an F# function is the return value, the sequence is returned from this function.

To write strings to a file as separate lines, you would use code like:

let WriteLines fileName (lines: string seq) =
    use writer = File.CreateText fileName
    for line in lines
        do writer.WriteLine line
This function takes a fileName and a sequence of lines and writes them into a file. (The parameter for the sequence of lines is contained in parentheses because the code is assigning a type of sequence of strings to the parameter lines. If this wasn't in parentheses, each word would be considered a parameter of the function.)

This is all great, but not very useful. To make sure I know how to employ file IO for a useful purpose, I created a little program to read the lines from a file, find all the lines that contain a word, and print them to the console output. Below is the program I created:

open System.IO

let readLinesFromFile filename =
    seq { use reader = File.OpenText filename
          while not reader.EndOfStream
              do yield reader.ReadLine () }

let CheckPhrase (x: string) =
    x.Contains("test")
    
[<EntryPoint>]
let firstMain argv =
    let fileLines = readLinesFromFile "testfile.txt"
    
    fileLines
        |> Seq.filter CheckPhrase
        |> printfn "Line with test -> %A"

    0

This program utilizes the functions I described above, plus one additional function. The additional function, CheckPhrase, takes a string parameter and returns true or false depending on when the input string contains the word "test." The other tidbit that is different, is the use of the Seq.filter function and the |> operator. The Seq.filter function creates a new sequence by passing one sequence (fileLines) through a "filter" function, in this case CheckPhrase. The the Seq.filter function adds an item from the first sequence into the new sequence if the passed in predicate returns true, otherwise it skips the element.

The other new piece of syntax is the pipe operator: |>. This operator "pipes" the output of the expression on the left to the expression on the right. In my program above, I am piping the fileLines variable into the Seq.filter CheckPhrase expression, whose result is piped into the printfn function.

Learning how to work with files in a new programming language is a way to gain practical experience with a new language. It also tends to expose new constructs or paradigms of the language to learn.

Wednesday, March 25, 2020

Learning F# - Step 3

This next step for me is an important one. Unit testing. I use unit testing, especially test driven development, a lot. For me TDD is a design tool that helps me design good code. Unit tests provide an awful lot of feedback about code that helps a developer learn, grow, and get better. (I'll try to create a separate post about the benefits of unit testing.)

All that being said, I like to know how to approach unit testing in a new language. Many people like to go after language constructs and idiomatic patterns first. These are important, but I like knowing how to test my code first. I won't always write unit tests as I'm learning and exploring, but understanding the approach to testing in a new language uncovers a LOT about the language itself.

F# makes unit testing easy, for a C# developer. F# supports NUnit and xUnit. I've been using xUnit for most of my C# projects, so I decided to try xUnit with F#.

To get started, I created a unit test project in F# using the following call in the .NET CLI:

dotnet new xunit --language f#
This call created a unit test project with the xUnit reference baked in. I then modified the Tests.fs file to be the following:
module Testing

open System
open Xunit

let AddOne x =
    x + 1

[<Fact>]
let TestAddOne () =
    let result = AddOne 5
    Assert.Equal(result, 6)

I immediately see some things that are familiar, and some things that are new. The module definition seems very much like the namespace structure from C#. Turns out, it is very similar. The module keyword packages the code that follows (because F# uses indentation and spacing instead of curly braces to separate code) into a coherent unit, just like the namespace keyword does in C#. The open System and open Xunit lines also look like import statements in C#. These are just like import statements from C#! The other familiar elements are the [<Fact>] attribute and the Assert.Equal() call. These are familiar elements of XUnit and work as expected.

To exercise my test, I created a function, AddOne, that takes a parameter and adds 1 to it. My test calls AddOne and captures the return value in the variable result. I then assert that my result is what I expect it to be. When I run the command:

dotnet test
I see that my unit test passes!

If you are following along at home, you've noticed that the default contents of the Tests.fs file is different from mine. It is useful to look at it as well. The default file contains a sample unit test with a strange name. The name is contained within a set of ``. This construct turns the contents contained within the `` set into an identifier that would not normally be a legal identifier. This "trick" allows one to create names in F# that can be used as a Domain Specific Language (DSL) or to create more readable output. That's a handy feature when doing a lot of unit testing!

Now that I have created my Hello World program, understand the build and project structure, and can create some basic unit tests, I'm itching to do something real with the language. That's next!

Monday, March 23, 2020

Learning F# - Step 2

My second step in learning a new programming language is understanding the build system for the project: how to get an actual working program from my code.

My approach to this step for F# in particular is interesting. It isn't interesting because I don't understand it. It is interesting because I have an emotional reaction to it. I'll explain that later.

The fundamental component of the F# build system is the FSPROJ file. This is automatically created when you use the CLI to create a new project. An FSPROJ file is just an XML file telling the .NET SDK which SDK to use, which files to compile, and what other packages are needed for this project to build correctly. Below is the default FSPROJ file from the default Hello World project:

<project sdk="Microsoft.NET.Sdk">

  <propertygroup>
    <outputtype>Exe</outputtype>
    <targetframework>netcoreapp3.1</targetframework>
  </propertygroup>

  <itemgroup>
    <compile include="Program.fs">
  </compile></itemgroup>

</project>

The first thing to know about F# project files is that they are part of MSBuild. MSBuild is the build utility that is used for C# and F# projects in the .NET space. To really dig into the project file, one needs to fully understand MSBuild. This post will not cover MSBuild.

The basics of the FSPROJ file are pretty easy to understand. The very first line specifies the SDK that should be used when building this project. The default entry will use the "default" SDK that you have specified on your workstation. (For the purposes of getting introduced to the language, this is fine.) Next up in the file you have the <PropertyGroup> tag. This grouping of elements tells the SDK what type of output to create (i.e. an executable, class library, etc.) along with the version of the framework to target. (If you are familiar with C# projects, you realize that there are many other options for this grouping. For the purposes of starting with F#, this is sufficient.) The last grouping is the <ItemGroup> section. This section lists the files that are included in this particular project.

I noticed that the FSPROJ file differs from the C# CSPROJ file in that the F# file lists the code files that are included in the project, where a CSPROJ does not. It turns out there are some peculiarities with having multiple F# code files in a project. I have not yet gotten that aspect figured out, so I will plan on an additional post once I understand that better.

The other oddity that I've noticed is that the only difference that I can see on the surface between an FSPROJ file and a CSPROJ file are the names (although the paragraph above points to another difference). I'm not sure why the two languages have differently named project files that come from the same build system. A quick internet search didn't reveal an obvious answer, so I'm still trying to figure that out.

And here begins my emotional reaction. <rant>The following paragraph is a mini-rant. Feel free to skip to the end.</rant> I am still getting over the removal of project.json from .NET Core! I completely understand why they did it, but I don't like, and I don't think I agree with it. Had they left project.json, it could have been a universal project file for both C# and F#. How cool would that be?!? Ok. I'm done.

For a basic application, the default project file works well. Most of the interaction I'm going to have with it is through the CLI tools for building, running, and adding references to my projects. This cursory understanding is enough to dive into the other areas of learning, but I know where to look if I need to do something more sophisticated.

Saturday, March 21, 2020

Learning F# - Step 1

With COVID-19 forcing everyone inside and cancelling events, I suddenly find myself with a lot of extra time. For the first few days, I just enjoyed having my weeknights free. Now, I want to use that time to learn something new. There are several things that I may bite off, but I've decided to start by learning F#.

F# is the functional programming language created by Microsoft. I was surprised to find out that it is baked into the .NET SDK that you download to create .NET Core applications! That fact is what prompted me to learn F#.

To get started with F#, you need to download the latest .NET Core SDK. Once you have .NET Core installed, you can get started!

Before I dive in, I wanted to share a little bit about how I approach learning a new programming language. I like to learn technology by doing things. With programming, I like to write code that solves problems. Quite often they are problems that I've solved over and over again, but they are still problems. When I approach a programming language, I want to do the following things:

  1. Create a Hello World program.
  2. Understand how projects are set up in the language.
  3. Learn unit testing in the language.
  4. Learn to work with the file system in the language.
  5. Learn to work with a database in the language.
  6. Learn to make web API calls in the language.
  7. Start learning the "particulars" of the language.
Using this approach allows me to start using the new programming language for real-life tasks, which helps my brain learn better.

Now that we know what language we are going to learn and how we are going to learn it, let's get started!

I have the latest .NET Core SDK installed on my box. To create the "Hello World" program, create a directory named FSharpHelloWorld. Change directory into FSharpHelloWorld, then all I have to do is:

dotnet new console --language f#
This simple command-line call creates a basic F# program that prints "Hello World" to the screen. Unfortunately, Microsoft created the Hello World program for me. In the directory, you should see the following 3 files listed:
obj
FSharpHelloWorld.fsproj
Program.fs
The FSharpHelloWorld.fsproj file is the "project" file that tells the SDK what to build and how to build it. (I'll dive into this file more in my next post.) The obj directory holds build artifacts. The Program.fs file is the code file that will get compiled and run. Inside the Program.fs file, you should see:
// Learn more about F# at http://fsharp.org

open System

[<EntryPoint>]
let main argv =
    printfn "Hello World from F#!"
    0 // return an integer exit code
The words "main" and "argv" look familiar to me as a C# developer, but the rest is a little foreign. I assume the "printfn" is something like a Console.WriteLine statement. The comments give me hints about the other line of the program. Since I have to feel like I've written code, I modify the text after "printfn" to read "Hola Mundo!" Next, I run the program by issuing the following command in my folder:
dotnet run
Lo and behold, I see
Hola Mundo!
printed on the screen!

So, what did I really accomplish? First, I was able to set up the basic SDK and get a simple environment up and running. Second, I was able to generate a simple program and make it run. Third, I was able to modify the program and have it run without compilation or runtime errors. I would say I have accomplished a lot!

Future posts will track my progress learning F#!