Domain Centric Product Teams: Pulling a Reverse Conway Manoeuvre to deliver better Software Products

Modern software engineering is oriented towards building networked distributed features for a highly connected and web savvy customer base in varying contexts. Traditional team structures within the enterprise have evolved from technical SME cliques as engineers who “Ate lunch together wrote Software together”

Good product strategy requires thinking about product features are built by engineering teams because Conway’s Law drives the outcome – to build, maintain and change great functional products we need to deliberately fix the team organisation

Conway’s Law and Layered Architecture

Melvin Conway described a law where he states that the parts of a software system are directly proportional to the organisational structure

  • “Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.” – Melvin Conway [1]
  • “If you have four groups working on a compiler, you’ll get a 4-pass compiler” – Eric S Raymond [2]

We can think of the modern teams oriented around the customer and end-systems to deliver integrated products in the following manner (circa 2020)

Thus, broadly speaking, teams orient around technical layers due to the cohesiveness of the “technical domain” expertise & customer’s (software client) needs.

Screen Shot 2020-08-17 at 1.29.17 pm

 

 

If you have been part of  similar teams in the past you must be familiar with the “layered architectural style” that gives rise to this team

SOALayers

 

Technical Teams = Technical Product

While the layered teams do a great job of grouping the teams by technical expertise, business needs for new end-to-end features are contextual, especially in a large enterprise. Things are more contextual in the real-world

Screen Shot 2020-08-17 at 2.52.24 pm

While the layered team structure may work initially for one context, we find adding new features and domain contexts makes it harder to maintain software components across a layer. This leads to slower software delivery due to various reasons

It is becomes increasing difficult for a single team to know and own aspects of a technical layer used in various contexts

Screen Shot 2020-08-17 at 2.59.30 pm

Conway’s law: With layered teams, we build layered software components focussed on the technical correctness vs end-to-end functionality

Techno-functional Teams = Functional Products

Applying the “reverse Conway manoeuvre” – fixing our team organisation to shape our software we break the technical teams into smaller functional teams. This process naturally builds a more “Domain Centric Product team” which can focus on functional needs and stay true to the end-to-end feature

This organisation allows for a closer relationship across technology layers as we group experts in different technical domains to deliver a common outcome and if done well, allows a smallish technical team to own a contextual solution

Screen Shot 2020-08-17 at 3.10.07 pm

 

Pitfalls of Functional Teams

Functional teams are not the perfect solution. Functional teams are highly agile and can own a specific end-to-end solutions through the entire software lifecycle but they operate in their own bubble

When organisations have different “technical practices” within their IT with different software engineering standards (one for the Portal team, one for the Integration team, one for the CRM team etc) it becomes harder to enforce technical standards within a technical context when a member is “outsourced” to another team

Summary

Modern software engineering is oriented towards building networked distributed features for a highly connected and web savvy customer base in varying contexts. Traditional team structures within the enterprise have evolved from technical SME cliques as engineers who “Ate lunch together wrote Software together”

Good product strategy requires thinking about breaking up the technical groups into highly effective functional teams and keeping the “band together”  through  the  lifecycle  of the software  (the end-to-end product)

 

 

 

Is your API killing your customer experience: Know the Context Boundaries

APIs can be used to query information or command changes in system of record. Even the most efficient & fast real-time web service still struggles with having to wait for an “ack” (acknowledgement) leading to coupling between the invoker & provider. If this happens within a cohesive business function then the coupling is necessary, however this can be a showstopper if done across business contexts!

User Experience

APIs with requests waiting on responses are notorious for blocking consumer applications

Dealing with slow or unresponsive APIs can be tolerated in certain circumstances however band-aid solutions can end up hurting user experience and damage your product’s reputation, pushing customers away

Good API design can help remediate this but it requires upfront understanding of how the customer facing systems are interacting with the internal systems & what their needs are

For example, simply throwing a cache & reading slightly stale information might be sufficient in a Query context – but how do you fix a Command API that is slow or unresponsive?

We look at some key concepts to break apart interactions and solve for X

Know your Contexts

Imagine you are in the business of selling coffees to happy customers and all you need is a point-of-sale system (let us imagine you have an endless tap of fresh coffee). Your business is all about selling then and you only operate in a single context

You may also define this as “We interact with the customer in a single context”, but let me warn you – you might be stretching to definition of a customer here (at-least from a technology implementation standpoint). I would argue they are mere transactional contacts vs true customers in the system (we will see why later)

Order Context

Context Boundaries

Let us now imagine your coffee shop now need to send out birthday coupons to your customers via email/sms etc. You may need to even call them and wish them (if they permit you) and so on

Customer Management Context

This act of recognising someone who is transacting with you and asking them for their preferences, interacting with them, offering them rewards for being a customer etc is different from selling coffee – welcome to the Customer Management context

Okay so your business has two contexts, Selling Coffee and Managing Customers. Are they the same or different? What actions can you not do  in both without impacting efficiency? How about capturing information (even if it on a bit of paper) in the two contexts about a person?

Most of us, who have never run a busy business, mix the two and have things work in them interchangeably. This works most of the time but it can lead to serious issues down the  line if we do not draw boundaries and divide the actions and information across the contexts.

Business Contexts

Ever been to a coffee shop where you waited a long time because the 1 worker was busy on the phone talking to a customer about their birthday plans? The worker was doing their job well in the customer management context but failed in the selling / transactional context

Contexts help define the actions, language and information while boundaries help define interactions. For engineers, this is vital as it helps design the connectivity as we have to often chose between consistency of information or availability of a service within networked or distributed systems

So why could your API be killing your Customer Experience?

APIs designed without consideration of context boundaries can end up coupling two contexts. Imagine an API for Contact creation in CRM which adds direct dependency in our coffee shop example. This API is sending data in real-time between a Selling context and Customer Management context, fine and can block or error the selling transaction if there are network delays or data issues in the CRM record, not fine!

Naïve APIs can Couple

We need to carefully consider interactions between context and introduce non-blocking interactions (and anti-corruption layers) to let business happen. Make Systems Integration Great Again!

Summary
In summary, services within a context have high cohesion and services across contexts should have low coupling. Bad API design (no not the RAML/OAS stuff but the solid/dotted arrow stuff) can couple two independent business contexts

Use Domain Events instead

If we carefully model our interactions by consciously recognising our contexts and boundaries then we can built for business transactions more efficiently and open up possibilities for future connectivity and growth

Better Digital Products using Domain Oriented APIs: The Shopping Mall Metaphor

APIs are the abstractions over technical services. Good APIs mirror strategic thinking in an organisation and lead to better customer experience by enabling high-degree of connectivity via secure mechanisms

Too much focus is on writing protocols & semantics with the desire to design good APIs and too little on business objectives. Not enough questions are asked early on and the focus is always on system-system integration. I believe thinking about what a business does and aligning services to leads us to product centric thinking with reuseable services

History
As an ardent student of software design and engineering principles, I have been keen on Domain Driven Design (DDD) and had the opportunity to apply these principles in the enterprise business context in building reusable and decoupled microservices. I believe the best way to share this experience is through a metaphor and I use a “Shopping mall” metaphor with “Shops” to represent a large enterprise with multiple lines of businesses and teams

Like all metaphors – mine breaks beyond a point but it helps reason about domains, bounded contexts, APIs, events and microservices. This post does not provide a dogmatic point-of-view or a “how to guide”; rather it aims to help you identify key considerations when designing solutions for an enterprise and is applicable upfront or during projects

I have been designing APIs and microservices in Health and Insurance domains across multiple lines of business, across varying contexts over the past 5-8 years. Through this period, I have seen architects (especially those without Integration domain knowledge) struggle to deliver strategic, product centric, business friendly APIs. The solutions handed to us always dealt with an “enterprise integration” context with little to no consideration for future “digital contexts” leading to brittle, coupled services and frustration from business teams around cost of doing integration ( reckon this is why IT transformation is hard )

This realisation led me to asking questions around some of our solution architecture practices and support them through better understanding and application of domain modeling and DDD (especially strategic DDD ). Thought this practice, I was able to design and deliver platforms for our client which were reusable and yet not coupled


Domain Queries 

In one implementation, my team delivered around 400 APIs and after 2 years the client has been able to make continuous changes & add new features without compromising the overall integrity of the connected systems or their data

Though my journey with DDD in the Enterprise, I discovered some fundamental rules about applying these software design principles in a broader enterprise context but first we had to step in to our customer’s shoes and ask some fundamental questions about their business and they way they function

The objective is to key aspects of the API ecosystem you are designing for, below are some of the questions you need to answer through your domain queries

  • What are your top-level resources leading to a product centric design?
  • When do you decide what they are? Way up front or in a project scrum?
  • What are the interactions between these domain services?
  • How is the quality and integrity of your data impacted through our design choices?
  • How do you measure all of this “Integration entropy” – the complexity introduced by our integration choices between systems?

The Shopping Mall example

Imagine being asked to implement the IT system for a large shopping complex or shopping mall. This complex has a lot of shops which want to use the system for showing product information, selling them, shipping them etc

There are functions that are common to all the shops but with nuanced difference in the information they capture – for example, the Coffee Shop does “Customer Management” function with their staff, while the big clothes retail store needs to sell its own rewards point and store the customer’s clothing preferences and the electronics retail does its customer management function through its own points system

You have to design the core domains for the mall’s IT system to provide services they can use (and reuse) for their shops and do so while being able to change aspects of a shop/business without impacting other businesses

Asking Domain and Context questions

  • What are your top-level “domains” so that your can build APIs to link the Point-of-Sale (POS), CRM, Shipping and other systems?
  • Where do you draw the line? Is a service shared by all businesses or to businesses of a certain type or not shared at all?
  • Bounded contexts? What contexts do you see as they businesses do their business?
  • APIs or Events? How do you share information across the networked systems to achieve optimal flow of information while providing the best customer experience? Do you in the networked systems pick consistency or availability?

Summary:

Though my journey with DDD in the Enterprise, I discovered some fundamental rules about applying these software design principles in a broader enterprise context. I found it useful to apply the Shopping Mall metaphor to a Business Enterprise when designing system integrations

It is important to understand the core business lines, capabilities (current and target state), business products, business teams, terminologies then do analysis on any polysemy across domains and within domain contexts leading to building domains, contexts and interactions

We then use this analysis to design our solution with APIs, events and microservices to maximise reuse and reduce crippling coupling

A Pandemic, Open APIs and Citizen Science: Its 2020 baby!

Human societies have been hit by pandemics through the ages and relied on the central governing authorities to manage the crisis and disseminate information. I believe this time around with COVID-19, our societies have access to more information from our governments because we have the internet

If this pandemic is an evolutionary challenge, then our response as a species to survive it will come through innovations in medicine, healthcare and technology. Not only will we improve on our lead time to develope vaccines as responses to viruses evolving and but also accelerate key technologies which will help us respond to global challenges as a whole

The internet has allowed governing agencies to share information the spread of COVID-19 in our communities through APIs a common channel in a clean, standardised, versioned, structured and self-describing manner leading to easier consumption by citizen and fuelling the rise of “citizen data scientists

I argue this democratisation of pandemic data via APIs and its consumption leads to new learning opportunities, increased awareness of the spread of the disease, verification of information, better social response and innovation through crowdsourcing

Open Data: NSW Health

The https://data.nsw.gov.au/ provides access to state health data in NSW Australia and in March 2020 provided information about COVID-19 cases on their site here. This website provides a very standardised approach to sharing this information, with metadata in JSON, RDF and XML for different consumers and links to the actual data within the metadata documents

Here is a screen shot of the actual data site

I particularly loved the structure of the JSON metadata because it is quite self-describing, leading to a link to the document with the COVID-19 data

Rise of the consumer: Our Citizen Scientist

It did not take long for someone to come along, parse that information and present to us in a portal we can all relate to. During the early days of the pandemic, I was hooked onto https://www.covid19data.com.au/ and it provided me with Australia / NSW wide information about the spread, categories etc

However it was Ethan’s site that I loved the most as a “local consumer” to see what is happening in my postcode – the website here https://covid19nsw.ethan.link/ is a brilliant example of the citizen science and is sourced from the NSW Health open data link above

Notice the URL and NSW Health agency did not built the portal – it was a this amazing person named Ethan

Summary

2020 is an interesting time in our history. We have the pandemic of our era but also better tools to understand its spread. The internet and standardised APIs are at the front and center of this information sharing and consumption

Everyone has the ability now to download information about the spread of the pandemic with time, geolocation and size embedded in this dataset. Everyone now have the ability to write programs to parse this dataset and do their own science on it

Stateful microservices pattern

What are stateful microservices?

Microservices holding state while performing some longer-than-normal execution time type tasks. They have the following characteristics

  1. They have an API to start a new instance and an API to read the current state of a given instance
  2. They orchestrate a bunch of actions that may be part of a single end-to-end transaction. It is not necessary to have these steps as a single transaction
  3. They have tasks which wrap callouts to external APIs, DBs, messaging systems etc.
  4. Their Tasks can define error handling and rollback conditions
  5. They store their current state and details about completed tasks

Screen Shot 2020-03-13 at 7.52.57 pm

Why stateful?

Stateless microservice requests are generally optimised for short-lived request-response type applications.  There are scenarios where long-running one-way request handling is required along with the ability to provide the client with the status of the request and the ability to perform distributed transaction handling and rollback (because XA sucked!)

So you need stateful because

  • there are a group of tasks that need to be done together as a step that is asynchronous with no guaranteed response-time or asynchronous one-way with a response notification due later
  • or there are a group of tasks where each step individually may have a short response time but  aggregated response-time is large
  • or there are a group of tasks which are part of a single distributed transaction if one fails you need to rollback all

Stateful microservice API

Microservices implementing this pattern generating provide two endpoints

  1. An endpoint to initiate: for example, HTTP POST which responds with a status code of “Created” or “Accepted” (depending on what you do with the request) and responds back with a location
  2. An endpoint to query request state: for example, HTTP GET using the process id from the initiate process response. The response is then the current state of the process with information about the past states

Sample use case: User Signup

  1. The process of signing-up or registering a new user requires multiple steps and interaction looks like this [Command]
  2. The client can then check the status of the registration periodically [Query]

Command

POST /registrations HTTP/1.1Content-Type: application/jsonHost: myapi.org

{ "firstName": "foo","lastName":"bar",email:"foo@bar.com" }
HTTP/1.1 201 Created  
Location: /registrations/12345

Query

GET /registrations/12345 HTTP/1.1Content-Type: application/jsonHost: myapi.org

{ "firstName": "foo","lastName":"bar",email:"foo@bar.com" }
HTTP/1.1 200 Ok  

{ "id":"12345", "status":"Pending", "data": { "firstName": "foo","lastName":"bar",email:"foo@bar.com" }}

Screen Shot 2020-03-13 at 7.38.41 pm

Anti-patterns

While the pattern is simple, I have seen the implementation vary with some key anti-patterns. These anti-patterns make the end solution brittle over time leading to issues with stateful microservice implementation and management

  1. Enterprise business process orchestration: Makes it complex, couples various contexts. Keep it simple!
  2. Hand rolling your own orchestration solution: Unlike regular services, operating long-running services requires additional tools for end-to-end observability and handling errors
  3. Implementing via a stateless service platform and bootstrapping a database: The database can become the bottleneck and prevent your stateful services from scaling. Use available services/products as they optimised their datastores to make them highly scalable and consistent
  4. Leaking internal process id: Your end consumer should see some mapped id not the internal id of the stateful microservice. This abstraction is necessary for security (malicious user cannot guess different ids and query them) and dependency management
  5. Picking a state machine product without “rollback”: Given that distributed transaction rollback and error-handling are two big things we are going need to implement this pattern, it is important to pick a product that lets you do this. A lightweight BPM engine is great for this otherwise you may need to hack around to achieve this in other tools
  6. Using stateful process microservices for everything: Just don’t! Use the stateless pattern as they are optimal for the short-lived request/responses use cases. I have, for example, implemented request/response services with a BPEL engine (holds state) and lived to regret it
  7. Orchestrate when Choreography is needed: If the steps do not make sense within a single context, do not require a common transaction boundary/rollback or the steps have no specific ordering with action rules in other microservices then use event-driven choreography

Summary

Stateful microservices are a thing! Welcome to my world. They let you orchestrate long-running or a bunch of short-running tasks and provide an abstraction over the process to allow clients to fire-and-forget and then come back to ask for status

Screen Shot 2020-03-13 at 8.37.14 pm

Like everything, it is easy to fall into common traps when implementing this pattern and the best-practice is to look for a common boundary where orchestration makes sense

Screen Shot 2020-03-13 at 8.33.59 pm

 

How did we get to Microservices?

If you have struggled with decisions when designing APIs or Microservices – it is best to take a step back and look at how we got here. It helps not only renew our appreciation for the rapid changes we have seen over the past 10-20 years but also puts into perspective why we do what we do

I still believe a lot of us cling to old ways of thinking when the world has moved on or is moving faster than we can process. APIs and microservices are not just new vendor-driven fads rather they are key techniques, processes and practices for businesses to survive in a highly evolving eco-system. Without an API-strategy, for example, your business might not be able to provide the services to consumers your competition can provide or with the quality of service we have come to expect (in real-time)

So, with that in mind let us take a step back and look at the evolution of technology within the enterprise and remember that this aligns to the business strategy

Long time ago

There were monolithic applications,  mainframe systems for example processing Order, Pricing, Shipment etc. You still see this monolith written within startups because they make sense – no network calls, just inter-process communication and if you can join a bunch of tables you can make a “broker” happy with a mashup of data

Screen Shot 2020-02-26 at 5.48.04 pm
1. First, there was just the legacy monolith application

Circa 2000s MVC app world

No ESB yet. Exploring the JVM and Java for enterprise application development. JSF was new and EJB framework was still deciding what type of beans to use. Data flowed via custom connectors from Mainframe to these Java applications which cached them and allowed viewing and query of this information

There were also functional foundation applications for enterprise logging, business rules, pricing, rating, identity etc. that we saw emerging and data standards were often vague. EAI patterns were being observed but not standardized and we were more focused on individual service design patterns and the MVC model

Screen Shot 2020-02-26 at 5.48.12 pm
2. Then we built lots of MVC applications and legacy monolith, integration was point-to-point

Services and Service-Oriented Architecture

The next wave began when the number of in-house custom applications started exploding and there was a need for data standardization, a common language to describe enterprise objects and de-coupled services with standards requests and responses

Some organisations started developing their own XML based engines around message queues and JMS standards while others adopted the early service bus products from vendors

Thus Service-Oriented Architecture (SOA) was born with lofty goals to build canonical enterprise data models, reduce point-point services (Java applications had a build-time dependency to services they consumed, other Java services), add standardized security, build service registry etc

We also saw a general adoption and awareness around EAI patterns – we finally understood what a network can do to consistency models and the choice between availability and consistency in a partition. Basically stuff known by those with a Computer Science degree working on distributed computing or collective-communication in a parallel computing cluster

One key observation is that the vendor products supporting SOA were runtime monoliths in their own right. It was a single product (J2EE EAR) running on a one or more application servers with a single database for stateful processes etc. The web services we developed over this product were mere XML configuration which was executed by one giant application

Also, the core concerns were “service virtualisation” and “message-based routing”, which was a pure stateless and transformation-only concept. This worked best when coupled with an in-house practice of building custom services and failed where there was none and the SOA product had to simply transform, route (i.e. it did not solve problems by itself as an integration layer)

Screen Shot 2020-02-26 at 5.36.36 pm
3. We started to make integration standardised and flexible, succeed within the Enterprise but failed to scale for the Digital world. Not ready for mobile or cloud

API and Microservices era

While the SOA phase helped us move away from ugly file-based integrations of the past and really supercharged enterprise application integration, it failed miserably in the digital customer domain. The SOA solutions were not built to scale, they were not built for the web and the web was scaling and getting jazzier by the day; people were expecting more self-service portals and XML parsers were response times!

Those of us who were lucky to let go off the earlier dogma (vendor coolade) around the “web services” we were building started realising there was nothing webby about it. After a few failed attempts at getting the clunk web portals working, we realised that the SOA way of serving information was not suited to this class of problems and we needed something better

We have come back a full circle to custom build teams and custom services for foundation tasks and abstractions over end-systems – we call these “micro-services” and build these not for the MVC architecture but as pure services. These services speak HTTP  natively as the language of the web, without custom standards like SOAP had introduced earlier and use a representational state transfer style (REST) pattern to align with hypermedia best-practices; we call them web APIs and standardise around using JSON as the data format (instead of XML)

Screen Shot 2020-02-26 at 7.01.52 pm
4. Microservices, DevOps, APIs early on – it was on-prem and scalable

The API and Microservices era comes with changes in how we organise (Dev-Ops), where we host our services (scalable platforms on-prem or available as-a-service) and a fresh look at integration patterns (CQRS, streaming, caching, BFF etc.). The runtime for these new microservices-based integration applications is now broken into smaller chunks as there is no centralised bus 🚎

Screen Shot 2020-02-26 at 7.08.20 pm
5. Microservices, DevOps, APIs on externalised highly scalable platforms (cloud PaaS)

Recap

Enterprise system use has evolved over time from being depended on one thing that did everything to multiple in-house systems to in-house and cloud-based services. The theme has been a gradual move from a singular application to a network partitioned landscape of systems to an eco-system of modular value-based services

Microservices serve traditional integration needs between enterprise systems but more importantly enable organisations to connect to clients and services on the web (cloud) in a scalable and secure manner – something that SOA products failed to do (since they were built for the enterprise context only). APIs enable microservices to communicate with the service consumers and providers in a standard format and bring with them best practices such as contract-driven development, policies, caching etc that makes developing and operating them at scale easier

De-mystifying the Enterprise Application Integration (EAI) landscape: Actors, terminology, cadence and protocols

Any form of Enterprise Application Integration (EAI) [1] work for data synchronization,  digital transformation or customer self-service web implementation involves communication between the service providers and service consumers. A web of connections grows over time between systems, facilitated by tools specialising in “system-integration”; this article covers how the clients, services and integration tools communicate and nuances around this observed in the wild

EAI Actors

Depending on the context, a system becomes a data-consumer or data-provider. These consumers and providers can be internal to a business enterprise or external to them. External providers can be pure software-as-a-service or partners with platforms and build teams entrenched in the “client-site”

Screen Shot 2020-02-25 at 3.41.20 pm
1. Services, Consumers within and outside a business enterprise

The provider and consumer systems are the key actors within an internal or external system-integration context. Cadences vary as some provider services are stable and mature, while others are developed with the client applications

Service/API Contract

Providers and Consumers communicate with each other using one of many standard protocols; also consumers have a direct dependency on service provider’s “service contract” to know about these protocols and service details

A service contract is a document describing one or more services offered by a service provider and covers details such as protocol, methods, data type and structure etc.

Screen Shot 2020-02-25 at 1.48.43 pm
2. Consumer, Service provider and Service Contract

A good service contract contains well-documented details of the service as well as “examples” of the request, response and errors. RAML [3] and Swagger/OAS [4] are two of the popular tools used in documenting service contracts

For example, the “Address search” contract by a SaaS vendor below describes the method, URI, query parameters and provides the user with the ability to “try out” the service. This approach allows consumers to iterate faster when developing solutions that use address search without having to engage the SaaS vendor teams (self-service)

Screen Shot 2020-02-25 at 3.49.51 pm
3. A Service Contract on an API Portal [2]
 

Service Contract Cadence: Contract Driven Development

Contract cadence is when a service contract is available compared to when the client wants to build their application. There are 2 cadences – mature and in-flight.

For a mature/pre-existing service, a good service contract allows the service consumer to develop a client application without having to engage a service provider person. Meanwhile, for a service being developed with the client application, there is an opportunity for both teams to author the service contract together during the elaboration phase of a project

Screen Shot 2020-02-25 at 4.29.52 pm
4. Deliver Cadence based on Service Contract Availability

Service contracts are key to consuming a service; when consuming mature services look for descriptive, sandbox-enabled, self-service services to supercharge your delivery. For services being developed in a project (along with the client applications), by internal teams or external vendors, ask for contracts upfront and ask for consumers & service providers to co-author the contracts to remove ambiguities sooner 

Avoid generating service contracts from developed components (Java Class to WSDL) as this technique leads to isolated, one-way, ambiguous specifications requiring considerable hand-holding and has the most defects during integration testing (from experience). Use Contract Driven Development [8] which facilitates the writing of a contract by the Service Provider and the Consumer together during Elaboration phase (sign in blood if adventurous)  

API Styles: RPC, REST,

Now what we have services, contracts out of the way we can dig deeper into how messages get over the wire in the real-time services. We will ignore “B2B protocols” for this discussion and leave them for the future

The four common real-time service protocols we see all are built over HTTP using  JSON or XML content-type and different in their implementation. Below is a short description of each

  • REST
    • Is the most service protocol for front-end applications and modern enterprise APIs. Is stateless, cacheable, uniform, layered etc and came from this [6]
    • A resource is a key abstraction in REST and a hard concept for integration-practitioners to master
    • Mature REST APIs look like Hypertext, i.e. you can navigate them like you would the web
    • Uses HTTP methods e.g. GET, PUT, POST, Patch for query and command
    • Uses HTTP status codes for communicating the response e.g. 2xx, 4xx, 5xx
    • Open to custom or standard request/response data-types. Standard hypermedia types include HAL, JSON API, JSON-LD, Siren etc. [4]
    • Requires resource-centric thinking
  • GrapQL
  • Streaming APIs
    • HTTP Streaming
    • Websockets
    • SSE
    • HTTP2/Push
    • gRPC
  • gRPC
  •  SOAP
    • Use to be popular before REST came along
    • Uses HTTP POST with the body containing request details – method-name, request etc in XML
    • Uses XML schemas for describing data types
  • Json-RPC
    • Semi-popular with some legacy clients
    • Uses HTTP POST with the body containing request details – method-name, request etc in JSON
    • Uses JSON object to define the method and data
  • OData
    • Used in CRM, ERP etc enterprise systems to reduce client-side development through config driven service consumptions
    • Presents the end service as a “data source” to the consumer, allowing SQL like query
    • Uses schema for describing data source objects and atom/XML  for transmitting over the wire
    • Requires custom parsers for parsing URL which contains the “query” to be mapped to the back-end service

There is plenty to cover in this space and in a future post, I will compare these protocols further and flavour them from experience. The key takeaway here though is that there are many many ways in which service provides and service consumers can communicate, most choose REST or SOAP over HTTP, there are passionate conversations over REST/SOAP, JSON/XML, HAL/JSON-API/Siren all the while OData remains a mystery to us -until we need to deal with it

EAI Patterns

There are heaps to learn about “how” these service providers and consumers communicate in a networked environment but below is a quick overview of these patterns. These patterns emerge because of the CAP Theorem [7] and the Network partition  between these systems looking to exchange data

Screen Shot 2020-02-25 at 5.57.40 pm
5. Integration Communication Patterns used by Service Providers and Consumers

Recap

  1. Enterprise Integration involves internal, external, batch and real-time services
  2. Key actors are service providers, consumers and mediators
  3. Service contracts are key documents in integrating
  4. The cadence between provider and consumer impacts delivery velocity
  5. Service protocols vary but there are 4-main types: REST, SOAP, JSON-RPC and OData

References

[1] EAI Patterns https://www.enterpriseintegrationpatterns.com/

[2] Experian REST API https://www.edq.com/documentation/apis/address-validate/rest-verification/#/Endpoints/Search_Address

[3] RAML https://raml.org/

[4] SWAGGER https://swagger.io/

[5] Choosing Hypermedia Formats https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/

[6] Roy Fielding’s dissertation https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

[7] CAP Theorem or Brewer’s Theorem https://en.wikipedia.org/wiki/CAP_theorem

[8] Contract Driven Development https://link.springer.com/chapter/10.1007/978-3-540-71289-3_2

[9] gRPC https://developers.google.com/protocol-buffers/docs/proto3

 

Tackling complexity: Using Process maps to improve visibility of integrated system features

“Entropy always increases– second law of thermodynamics

Enterprise systems are similar to isolated physical systems, where the entropy or hidden-information always increases. As the business grows, our technology footprint grows as new systems are implemented, new products and cross-functional features are imagined and an amazing network of integrations emerge

Knowing how information flows and managing the chaos is therefore critical organisations are to move beyond “Functional-1.0” into “Lean-2.0” and “Strategic-3.0”  in their implementations. We discuss how current documentation and technical registries simply “tick the box” and there is a better approach to manage increasing complexity through better context

Enterprise Integration Uses Integrations

Current state: Integration Interface Registries with little context

The network of integrations/interfaces (blue-circles above) are often captured in a technically oriented document called “Interface Registry” in a tabular form by teams performing systems integration. While these tables provide details around “who” (producer/consumer details) and “how” (the type of integration) they cannot describe “when” and “why” (use case).  As projects grow and interfaces grow or are re-used the number of when and whys increase over time and entropy (hidden information) around these interfaces grows; this leads to chaos as teams struggle to operate, manage and change them without proper realisation of the end-to-end picture

As a result, only maintaining a technical integration Interface registry leads to poor traceability (business capability to technical implementation), increased maintenance-cost of interfaces ( hard to test for all scenarios) and leads to duplication of effort over time ( as change becomes complex, teams rewrite)

Screen Shot 2019-11-21 at 6.22.21 pm

Integration Interface Repository

Therefore without proper context around Integration Interfaces, organisations will struggle to manage and map cross-functional features leading to slower lead-time, recovery etc over time. We propose that documenting integration use-cases, in a business-friendly visual language and related them to technical interface lists and enterprise capabilities is the key to mastering the chaos

Mastering the chaos: Building a context map

Context is key as it

  1. It drives product-centric thinking vs project-based thinking
  2. It makes our solution more operable, maintainable and re-useable 

In order to  provide better context and do it in a clear visually-oriented format, we believe documenting integration user-stories as technical process flows is a good start

Consider the following use-case: “As a user, I must be able to search/register/update etc in a system”.  Use-cases begin all start with some activation point – a user, timer or notification and then involve orchestration of services or choreography of events resulting in actions within microservices or end-systems eventually delivering some value through a query or command. We can render such a use-case into a map showing the systems, interfaces and actions in them (activation point, services, orchestrations, value) and do so in a standard manner

Screen Shot 2019-11-21 at 5.58.33 pm

For example, we leveraged the Business Process Management Notation – BPMN 2.0 standards to map integration technical use-case flows where we used general concepts like “swim-lanes” for user and systems, “arrows” for Interfaces (solid for request-response interfaces, dotted-lines for async messages) etc.

The Picture below shows this concept along with the “Interface” lines and “Messages” connecting the boxes (actions) between systems. Each interface or message then was linked to the Integration Interface Registry so that it was easy to trace reuse and dependencies

Screen Shot 2019-11-21 at 4.26.04 pm

It was also important that the context picture above is fairly lean as it avoids documenting too much to avoid becoming a single giant end-to-end picture with everything on it. It is best to stay within a bounded-context and only refer to a specific use-case such as “User Registration” or “Order submission” or “Customer Management” etc. This has the added advantage of helping teams which speak a ubiquitous language talk to a collection of pictures belonging to their domain and integration-practitioners to identify a collection of such teams (bounded-contexts)

Building a library and related to EA

The journey to improve visibility and maintenance of integration artefacts then involves capturing these integration use-case context maps, storing them in a version-controlled repository, relating them to other technical and business repositories

This collection of context maps would contain similar information to a “high-level enterprise system integration view” but with a greater degree of clarity

Screen Shot 2020-02-24 at 7.02.43 pm

This collection can also be linked to the Enterprise Architecture (EA) Repository for full end-to-end traceability of Business Capabilities into Technical Implementations. In fact, the TOGAF framework describes an external Business Architecture repository pattern as part of Solution building blocks (see TOGAF structural framework )

We imagine the Integration Context Map repository linked to the Enterprise Architecture Repository and the Integration Interface repository as shown below – this would provide immense value to cross-functional teams and business stakeholders, allowing both to see a common picture

Screen Shot 2019-11-19 at 7.54.49 pm

Sequence flows or process flows?

Sequence diagrams can also be used to document technical use-cases with systems and interfaces, however similar to the Integration interface list, they then to be difficult to consume for the non-technical users and lack the clarity provided by process maps

Screen Shot 2019-11-19 at 6.06.42 pm

As a general rule of thumb we found the following segregation to be useful:

  1. What: Technical process flows for end-to-end visibility, especially useful in complex long-running distributed features.  Sequence diagrams for technical component designs, best for describing how classes or flows/sub-flows (in Mule, for example) interact
  2. Who:  Context maps by Business Analysts (BA) or Architect and Sequence flows by Developers
  3. When: Context maps by Business Analysts (BA) as early as during project Discovery, providing inputs to sizing and visual map of what-is-to-be (sketch?). Sequence flows by Developers, as a task in Development story
Screen Shot 2019-11-14 at 6.43.16 pm

Let us talk tools

There are a variety of tools that can help document process context maps in a standard BPMN 2.0 format. The key criteria here is to produce a standard artefact – a BPMN 2.0 diagram so that it can be managed by standard version-control tools and rendered to documents, team wikis etc. though tools/plugins

Below is a list of tools you can try, we recommend not getting too hung up on tools and instead focus on the practice of documenting integration use-cases

Tools

Recap

  1. As enterprise projects deliver more integrated solutions, it becomes harder to manage and change integration interfaces without proper traceability
  2. Improve traceability of a single end-to-end use-case through a context map
  3. You can use BPMN 2.0 for a standardised notation to do this and use tools to generate these context maps as .bpmn files
  4. You can version control these .bpmn  files and build a collection of context maps
  5. You can link these context maps to Integration Interface registry and Enterprise Business capability registry for increased traceability across the enterprise
  6. There are many tools to help you write the .bpmn files, don’t get hung up on the tools. Start documenting and linking to the interface registry

Conclusion

The context map collection then becomes very useful for enterprise architecture, integration operations, new project teams, testing etc. as a common visual artefact as it relates to the users, systems and interfaces they use 

Enterprise Integration process maps then become a powerful tool over time as they greatly improve visibility across the landscape and help teams navigate a complex eco-system through a contextual and meaningful visual tool; this leads to better open and maintainable integration products leading to reuse and cost-efficiency 

 

Why is this not an API contract?

Why is this … my Swagger UI, generated from code not a contract? It describes my service, therefore it must be a Service Provider Contract. No? 

This was a common theme for a few of our clients with mobile/web teams as consumers of enterprise services.  Service providers generated contracts, and would sometimes create a contract in the API portal as well.

Service consumers would then read the contract from the API portal and consume the service. What the problem then?  …

SwaggerGenWhatIsIt.png

….the problem is that the red box i.e the Contract – is generated after the service is implemented and not vice-versa.

Why is this a problem then?  It is a problem because the contract is forced upon the consumer and worse there are 2 versions of this document.

So what? Well as you can imagine, changes to the service implementation over time will generate the provider contract (red box), while consumers continue to read the out-of-sync contract.

so? A contract is an agreement between the two parties – consumer and provider. In the above use-case though, this is not the case.

Key Points:

  • Generated Swagger UI is a documentation and not a contract
  • A contract is a collaborative effort between Providers and Consumers
  • A product (API Gateway) cannot solve this problem, it is cultural
  • The above process will create 3 layers of responsibility – Service provider, Service consumer and middleware provider
  • The 3 layers of responsibility makes it harder to test APIs

Side note: I believe this was a big problem with SOA – the “Enterprise Business Service (EBS)” was owned by the middleware team and “Application Business Services (ABS)” was owned by the services teams.

The fix?

Collaborative contracts that help define what a service should do!

This contract is used by Consumers to build their client-code and more importantly the providers use the contract to build the service and test it!

Contract.png

 

 

Lessons from API integration

A general transition has been happening with the nature of work that I do in the integration space … we have been doing less of SOAP/XML/RPC webservices and more RESTful APIs for “digital enablement” of enterprise services .  This brought a paradigm shift and valuable lessons were learnt (rightly or wrongly) … and of course the process of learning and comparing never stops!

Below are my observations …

It is not about SOAP vs REST … it is about products that we integrate, the processes we use and the stories we tell as we use one architectural style vs the other

  1. Goals: Both architectural styles work to achieve the same goal – integration. The key differences lie in where they are used. SOAP/XML has seen some b2b but adoption by web/mobile clients is low
  2. Scale: SOAP/XML is good but will always remain enterprise scale … REST/JSON is webscale. What do I mean by that? REST over HTTP with JSON feels easier to communicate, understand and implement.
  3. Protocol: The HTTP protocol is used by REST better than SOAP, so the former wins in making it easy to adopt your service
  4. Products: The big monolith integration product vendors are now selling “API Gateways” (similar to a “SOA product suite”) … The gateway should be a lightweight policy layer, IMHO, and traditional vendors like to sell “app server” licenses which will blow up an API gateway product (buyer beware!)
  5. Contract: RAML/YAML is a lot easier to read than a WSDL ….which is why contract first works better when doing REST/JSON
  6. Process: The biggest paradigm change we have experienced is doing Contract Driven Development  ny writing API definition in RAML / YAML … Compare this to generating wsdl or RAML from services. Contract driven , as I am learning, is much more collaborative!
  7. Schemas: XML schemas were great until they started describing restrictions … REST with JSON schema is good but may be repeating similar problems at Web scale
  8. Security: I have used OAuth for APIs and watched them enable token authentication and authorization easily with clients outside the enterprise – we are replacing traditional web session ids with oauth tokens to enable faster integration through 3rs party clients … all this through simple configuration in an API gateway compared to some of the struggle with SAML setup within an organisation with the heavy monolithic SOA and security products!

It would be easy to then conclude that we ought to rip out all our existing SOAP/XML integrations and replace them with REST APIs no? Well not quite…as always “horses for the courses”.

Enterprise grade integration may require features currently missing in REST/JSON (WS-* , RPC) and legacy systems may not be equipped to do non XML integration

My goal was to show you my experience in integrating systems using API with Contract driven development & gateways with policies vs traditional web service development using SOA products …hope to hear about your experience, how do you like APIs?