Psychosomatic, Lobotomy, Saw

FIX protocol messages are divided into two types.

1. Session Layer Messages
2. Application Messages

Session Layer Messages are for establishing, recovering and terminating FIX session. While application messages are to carry out business purposes such as buy, sell, sending/subscribing market data or sending execution reports.

FIX protocol session layer is a realization of layer 5 session layer of OSI model.

Image Reference - Data Networks - MIT

FIX session can exist multiple sequential FIX connections. FIX connection can be terminated due to various reasons such as application or network outages. But the FIX session can continue after re-establishing the FIX connection. There are various means provided in the FIX session layer specification to recover the session as well. Such as keeping track of NextNumIn, NextNumOut, Retransmission, Gap-Fill and Resend of messages.

Image Reference - Session Across Sequential Connections

Establishing FIX Connection

Establishing FIX connection has three parts.

1. Transport Layer Connection
2. Acceptance with optional authentication
3. Message synchronization

FIX session time-span

FIX session usually exists for agreed time-span by two peers (acceptor and initiator). It can be weekly or daily sessions. This weekly and daily session timing are agreed by out-of-band communications. Which means no session layer message will specify the duration of the session. Which we called the in-band communication. Two counterparties agree upon session timing as part of their rules of engagement.

Since I briefly mention in-band communication. Example for in-band communication is the heart beat interval. Which used by peers to check on each other’s live-ness. HeartBtInt(tag 108) is set in initiators logon request (35=A) to specify this value. Hence, its in-band communication.

Important out-of-band parameters

Following out-of-band parameters are important to the session layer.

Heartbeat interval determination

In QuickFIXJ initiator will specify the HeartBtInt(108) tag in the logon request (35=A). Acceptor which may or may not be QuickFIXJ based implementation, will also send HeartBeatInt(108) in its logon request. Acceptor may echo back the value send by the initiator, or send a different value. Or it can even reject the logon request. Initiator can also reject the logon message send by acceptor if it does not agree with the amend for the HeartBeatInt.

QFJ Configuration - Initiator

Following is a QFJ log from the perspective of the accepting peer. HeartBtInt configuration is mandatory for QFJ initiator peer. Acceptor will switch to the heart beat interval set by the initiator in the logon request.

Above screenshot is from the output of the qfj-fix-shell

Message Recovery

FIX message synchronization after logon request has been send considered to be recoverable if the followings are true.

peer1.NextNumOut >= peer2.NextNumIn
peer2.NextNumOut >= peer1.NextNumIn

What this means is peer’s NextNumOut should be always equal or smaller than other peer’s expected NextNumIn. If there is need for message synchronization, then it can be done using ResendRequest(35=2) and SequenceReset(35=4) messages which follows the Logon(35=A) messages.

Scenario 1 - Acceptor requires retransmission of messages from initiator

Let’s take a scenario where Initiator’s NextNumOut is larger number than the Acceptor’s NextNumIn.

    initiator.NextNumOut > acceaptor.NextNumIn 
    initiator.NextNumIn = acceaptor.NextNumOut

Above figure shows a scenario I have created using qfj-fix-shell. Where initiator’s NextNumIn and acceptor’s NextNumOut matches(83), but initiator’s NextNumOut (199) is larger than acceptor’s NextNumIn (20).

Let’s assume on the initiator side message sequence for un-synced outgoing messages would look like this.

Recovery process

Scenario 2 - Initiator requires retransmission of messages from acceptor

Let’s take a scenario where Initiator’s NextNumOut is larger number than the Acceptor’s NextNumIn.

    initiator.NextNumOut = acceaptor.NextNumIn 
    initiator.NextNumIn < acceaptor.NextNumOut

Above figure shows a scenario I have created using qfj-fix-shell. Where initiator’s NextNumOut and acceptor’s NextNumIn matches(121), but initiator’s NextNumIn (50) is smaller than acceptor’s NextNumOut (122).

Let’s assume on the acceptor side message sequence for un-synced outgoing messages would look like this.

Recovery process

Scenario 3 - Both initiator requires retransmission of messages from acceptor

Let’s take a scenario where Initiator’s NextNumOut is larger number than the Acceptor’s NextNumIn.

    initiator.NextNumOut > acceaptor.NextNumIn 
    initiator.NextNumIn < acceaptor.NextNumOut

Above figure shows a scenario I have created using qfj-fix-shell. Where initiator’s NextNumOut(59) is larger than acceptor’s NextNumIn(46), and initiator’s NextNumIn (43) is smaller than acceptor’s NextNumOut (59).

Let’s assume on the acceptor side message sequence for un-synced outgoing messages would look like this.

Let’s assume on the initiator side message sequence for un-synced outgoing messages would look like this.

Recovery process

Best Practices

Always send a TestRequest(35=1) upon Login(35=A)

Initiator and Acceptor require to perform message synchronization after each peer received the logon request, and logon request SequenceNumber does not match with the expected NextNumIn for the respective peer. A good practice is to send TestRequest(35=1) to force message synchronization the peer to send a HeartBeat(35=0), before sending any queued application level messages.

References

FIX Session Layer Test Cases

FIX Session Layer Complete References

Agrona is a compilation of threading and data structures which essentially a utility library, which is the underpinning of Aeron, SBE and Artio. These are all parts of the real-logic stack for building high performance financial applications.

AgentRunner, Agent, Duty-Cycle and the Idle Strategy

Let’s visualize a new approach of writing business logic. Your logic resides in an Agent class. Which has a method doWork() that get executed in a loop. Each iteration is called a Duty Cycle. And each Agent is managed by an AgentRunner. AgentRunner is essentially a Runnable. Using AgentRunners static method, AgentRunner can be executed in a thread using default non-daemon ThreadFactory.

The thread will run following piece of code in an infinite loop. Until AgentRunner is closed.

final int workCount = agent.doWork();
idleStrategy.idle(workCount);

doWork() method returns an integer value, the idiomatic usage of this return value is to indicate the number of work carried out by the Agent. For an example number of network messages handled by the Agent during the Duty Cycle. Idle Strategy is used by the AgentRunner to decide take breaks between Duty Cycles. If during the previous Duty Cycle doWork indicate no work has been done, then the Idle Strategy will put the thread in to sleep.

public void idle(int workCount) {
    if (workCount <= 0) {
        LockSupport.parkNanos(this.sleepPeriodNs);
    }
}

The rationale behind this approach is to free up the CPU cores if application has no new events to execute. Then release the CPU core by putting the thread in to sleep. This is what we called the voluntary context switching. It may be not the best choice of action in terms of the particular application. But it’s good for overall system performance, where multiple applications run’s on same OS host.

Simple example

We have simple Agent implementation called RandomWorkCountAgent which simulate Agent with duty cycles which work has been carried out and also duty cycles where no work has been carried out.

import org.agrona.concurrent.Agent;
import java.util.Random;

public class RandomWorkCountAgent implements Agent {

    private final Random random = new Random();

    @Override
    public void onStart() {
        System.out.printf("[%s], Agent started\n", Thread.currentThread().getName());
    }

    @Override
    public int doWork() throws Exception {
        return random.nextInt(0, 3);
    }

    @Override
    public void onClose() {
        Agent.super.onClose();
    }

    @Override
    public String roleName() {
        return "random-agent";
    }
}

Idle strategy implementation is LogSleepIdleStrategy where is work count is zero it will make the thread sleep for 2 seconds. And if work count is greater than zero, log the work count and return.

import org.agrona.concurrent.IdleStrategy;

import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.LockSupport;

public class LogSleepIdleStrategy implements IdleStrategy {
    @Override
    public void idle(int workCount) {
        if (workCount <= 0) {
            System.out.printf("[%s], Going for a sleep\n", Thread.currentThread().getName());
            LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(2L));
        } else {
            System.out.printf("[%s] - work count = %d\n", Thread.currentThread().getName(), workCount);
        }
    }

    @Override
    public void idle() {
    }

    @Override
    public void reset() {
    }

    @Override
    public String alias() {
        return IdleStrategy.super.alias();
    }
}

Main program.

import org.agrona.concurrent.AgentRunner;

public class SimpleAgentSample {
    public static void main(String[] args) {
        AgentRunner agentRunner = new AgentRunner(new LogSleepIdleStrategy(),
                throwable -> {}, null, new RandomWorkCountAgent());
        AgentRunner.startOnThread(agentRunner);
    }
}

Once you run the program you may observe something similar to bellow output. Where Idle strategy logs work count or indicate thread sleep. In duty cycles where work count is zero. Thread sleep before execute the next duty cycle.

[random-agent], Agent started
[random-agent] - work count = 1
[random-agent] - work count = 2
[random-agent], Going for a sleep
[random-agent] - work count = 1
[random-agent] - work count = 2
[random-agent] - work count = 2
[random-agent] - work count = 2
[random-agent] - work count = 1
[random-agent] - work count = 1
[random-agent] - work count = 2
[random-agent] - work count = 1
[random-agent] - work count = 2
[random-agent], Going for a sleep
[random-agent] - work count = 1
[random-agent], Going for a sleep

Is this a sensible approach

Answer to that question simply is NO. Our objective is to process events as fast as possible. Just because a duty cycle reported no work has been done, does not imply that next duty cycle will also have no events or work to be done. By making thread move to voluntary context switch we waste lot of CPU cycles and reload the CPU instructions. But it may be also impossible to predict how work count in one duty cycle could be used to predict the work count for next duty cycle.

There are many Idle wait strategies Agrona provide to cater different use cases. You can use them to be more CPU conservative or less CPU conservative. Choice is really up to you. Each come with their own perks.

However, decoupling the CPU conservativeness from business logic in my opinion is what makes Agrona Agent’s standout among other threading libraries.

Writing Custom Idle Wait Strategy

Imagine extra CPU conservative idle strategy, in which the idle time exponentially increase with more and more duty cycles reporting no work has been done. This strategy is a suitable candidate for network reconnection, or low throughput high latency message processing.

As illustrated bellow, after first duty cycle with zero work count, thread sleep for 100ms, and as duty cycles keep producing zero work count. The sleep time increase to 200ms, 400ms. We usually cap the maximum wait time as the wait time will increase to very large value by few iterations. And that big of a wait is meaningless.

import org.agrona.concurrent.IdleStrategy;
import java.util.concurrent.locks.LockSupport;

public class ExponentialBackOffIdleStrategy implements IdleStrategy {

    private final long maxBackOff;

    private final long initialBackoff;

    private long backoffCounter;

    private long currentBackOff = 0;

    public ExponentialBackOffIdleStrategy(long initialBackoff, long maxBackOff) {
        this.maxBackOff = maxBackOff;
        this.initialBackoff = initialBackoff;
    }


    @Override
    public void idle(int workCount) {
        if (workCount > 0) {
            backoffCounter = 0;
            currentBackOff = 0;
        } else {
            if (currentBackOff < maxBackOff) {
                long sleepTime = initialBackoff * Math.round(Math.pow(2, backoffCounter));
                backoffCounter++;
                currentBackOff = Math.min(sleepTime, maxBackOff);
                System.out.printf("A new backoff calculated %dms\n", Math.round(currentBackOff * 1e-6));
            }

            LockSupport.parkNanos(currentBackOff);
        }
    }

    @Override
    public void idle() {
        LockSupport.parkNanos(this.initialBackoff);
    }

    @Override
    public void reset() {

    }

    @Override
    public String alias() {
        return IdleStrategy.super.alias();
    }
}

GitHub-pages is a way of hosting Jekyll based static sites. I decided to use it to host my tech blog. After all we are all living in the age of everything as code.

So why not blog as code.

This post is a TL;DR and a compilation of sources for more details.

Specification for a blog solution

1. Should have ability to run locally and test before publishing (offline support)
2. Version control
3. Use markdown as the formatter for writing
4. Have good templating support

Solution 1 - Self Hosting

Jekyll is site generator where you can write content for your site using markdown format. There are many themes available as well, to bootstrap your site code. I used to host my blog by running jekyll server on Amazon EC2 instance and using AWS Route 53 Authoritative DNS server to route traffic to the public IP of my EC2 instance. To enable TLS I used Let’s Encrypt to get a server certificate which is valid for 3 months. And use the certbot to renew my server certificate automatically.

That’s a lot of work, also recurring cost for AWS infrastructure.

Solution 2 - Using GitHub Pages

You have to create a new github repository with name .github.io.

For an example, my GitHub username is busy-spin. And the repository I use to host my blog is busy-spin.github.io. Publishing content to your website is as simple as pushing content to GitHub repository.

I can run the blog site locally as both approaches use Jekyll site generator.

Testing Site Locally

Bellow is a complete guide on how to test your site before publishing.

Guide - Testing your side locally with Jekyll

Important commands are

To install the required Ruby packages

bundle install

To run the jekyll server locally

bundle exec jekyll serve

Jekyll process has a file watcher so once you update the content it will auto compile it to html code.

Extras

Adding mermaid flow chart support for GitHub pages.

mermaid.js chart support is available by default in GitHub wiki, GitHub projects. But enabling mermaid.js for GitHub pages is not supported out of box and also not trivial to enable. I found this neat trick from fellow GitHub blogger jackgrubber - Embed-Mermaid-in-Jekyll-without-plugin.

```mermaid
flowchart TD
    A[Learn difficult concepts] -->|Create Experiment| B{Experiment}
    B --> | Too hard to understand | C[Read more and write about it]
    B --> | Got a good grip of concept | D[Write about it]
    C --> | Experiment Again | B
    D --> | Level up again | A
```  

flowchart TD
    A[Learn difficult concepts] -->|Create Experiment| B{Experiment}
    B --> | Too hard to understand | C[Read more and write about it]
    B --> | Got a good grip of concept | D[Write about it]
    C --> | Experiment Again | B
    D --> | Level up again | A

Other Helpful References

Jekyll Quick Start

GitHub Pages - Quick Start

Writing on GitHub - A complete guide on markdown

GitHub - Basic Writing And Formatting Syntax

TL;DR

As a developer I use Github to share my code with others. Being a java developer, I also see the advantage of using java libraries as a form of sharing our work. Once java library is developed with useful release features, it can be deployed to Sonatype maven central repository. Github offers the GitHub Actions where we can seamlessly automate from code commit to library publish.

How

1. Create sonatype account, get token to publish maven artefacts
2. Claim the namespace so you can publish under maven <groupId>
3. Create gpg keys, include gpg private key in your build and add the public key in to key server, this is to perform integrity checks for libraries you publish
4. Set up secrets for your repository including sonatype username, token, gpg private key, gpg passphrase
5. Set up GitHub action workflow so you can publish libraries from GitHub to Sonatype

Why

Claiming your namespace

All my libraries go under <groupId>io.github.busy-spin</groupId> I should claim this namespace in Sonatype. Pretty much similar to how you get a username in GitHub or blog-space.

Generate GPG private key and public key

Standard why to perform a integrity check for your libraries is to generate GPG key pair. Keep the private key with you (In this case keep it in GitHub Secrets). And publish the public key to gpg key server. When you publish the artefacts, the artefacts will be signed using your private key. And Sonatype server will verify using the published public key.

GitHub Actions / GitHub Secrets

As your library start getting traction, more and more people would want to collaborate with you. It’s easy to centralize the publishing instead of all contributors sharing GPG keys and other secrets and settings up the deployment related settings in their individual systems.

Steps

Create sonatype account and get token

Use following link to create account in Sonatype and create user token.

Generate Sonatype User Token

Claim your namespace

Use the following link to claim the namespace which is your maven <groupId>

Claim Namespace

Generate GPG key

Use following guide on generating gpg-key pair and publishing the public key to gpg key server

Generate GPG Keys

Use following command to copy your GPG private key, replace id with your GPG key pair id. GPG private key and GPG passphrase will be later uploaded in to GitHub secrets.

gpg --armor --export-secret-key <id>

Setting up secrets

Use following guide on more details on settings up secrets.

Adding Github Secrets

I need to configure four secrets to support the GitHub Actions that I will later set up.

GitHub acton to deploy to Sonatype

Set up GitHub Actions to deploy library to Sonatype

name: Publish package to the Maven Central Repository
on: [workflow_dispatch]
jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - name: Set up JDK 17
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Build with Maven
        run: mvn -B package --file pom.xml

      - name: Set up Apache Maven Central
        uses: actions/setup-java@v4
        with: # running setup-java again overwrites the settings.xml
          distribution: 'temurin'
          java-version: '17'
          server-id: maven # Value of the distributionManagement/repository/id field of the pom.xml
          server-username: MAVEN_USERNAME # env variable for username in deploy
          server-password: MAVEN_CENTRAL_TOKEN # env variable for token in deploy
          gpg-private-key: $ # Value of the GPG private key to import
          gpg-passphrase: MAVEN_GPG_PASSPHRASE # env variable for GPG private key passphrase

      - name: Publish to Apache Maven Central
        run:  mvn --batch-mode deploy -P publish-sonatype
        env:
          MAVEN_USERNAME: $
          MAVEN_CENTRAL_TOKEN: $
          MAVEN_GPG_PASSPHRASE: $

Update you pom.xml to include gpg and sonatype publisher plugins

    <profiles>
    <profile>
        <id>publish-sonatype</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.sonatype.central</groupId>
                    <artifactId>central-publishing-maven-plugin</artifactId>
                    <version>0.4.0</version>
                    <extensions>true</extensions>
                    <configuration>
                        <publishingServerId>maven</publishingServerId>
                        <tokenAuth>true</tokenAuth>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-gpg-plugin</artifactId>
                    <version>1.6</version>
                    <executions>
                        <execution>
                            <id>sign-artifacts</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>sign</goal>
                            </goals>
                            <configuration>
                                <gpgArguments>
                                    <arg>--pinentry-mode</arg>
                                    <arg>loopback</arg>
                                </gpgArguments>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

Execute the GitHub Action

Lastly login to Sonatype to publish the uploaded library.

Publishing Deployment

Refrences

Refer to my aeron-maven-plugins GitHub project.