Hello! This one is going to be short and less formal a post. I want to get these questions out there before they eluded me and then come back later to this post (or another) and answer some of these
I have been thinking about how we are putting out more integrated solutions now than 10 years ago and how two implementations with the same number of systems grow differently over time to be more or less nimble and more important grow to more or less chaos with issues around Data integrity, Operations etc
I want to apply formal analysis to compute the amount of “hidden information” (entropy) in our implementations both from the systems and integrations perspective and from the data flowing through it
It is like calculating how ordered the arrangement of a set of marbles will be bouncing over a platform supported by jointed arms
Source: https://www.nutsvolts.com/magazine/article/the_flying_marbellos
What’s with the Topic?
Okay. Entropy is close to me because I love Time Travel and in reality you cannot time travel become of the 3rd Law of Thermodynamics (darn you Boltzman) which says “Entropy Always Increases”
While entropy has to do with heat, it actually talks about the amount of order in a system and the fact that you cannot go back to order from a chaotic arrangement (broken cup) without expending energy
Thinking about this a little deep, if we consider the systems and integrations we build then there is a state where they are in perfect order and with each transaction integrated solutions move towards chaos (or disorder)
Fun Experiment: Try doing this with post-it notes and 2 / more kids as the nodes – ask one child to tell the other to do something and pass the notes as information etc. Things will get chaotic over time!
So what?
More nodes, edges, data in edges = high degree of freedom or possible states for the system to be in and this is the entropy. I argue that well integrated enterprise systems feel easier to manage and operate because they requires less energy to bring to order – this is for the integration platform. I am trying to think about what this means for the data that these integrations bring – one hypothesis is that the type of integrations determine the quality of data as coupling contexts could cause data chaos
Wait what?
Yep. I think given the same set of enterprise systems and same integration product – the initial conditions in data (what the systems have) and the integration design determine how much Data Entropy an organisation has over time. The Integration Entropy may be the same but the Data Entropy could be different
What’s next?
We want to come up with a way to measure this complexity for a proposed or existing solution
Here are some of my notes and questions, feel free to ping me your thoughts
- Integrations add complexity
- We are doing more integrations because the Web has come to the Enterprise
- 10 years ago we were struggling to deliver more than 5-10 end-to-end integrations in a 12 month period due to environments, team structure, protocol issues etc. We have accelerated with DevOps, PaaS, APIs, Contracts and better tools to deliver end-to-end solutions ( we built 100 interfaces in 10 months )
- More end-to-end contextual micro-services
- More moving parts
- Greater degree of freedom in which clients use these services and integrate
- What is the amount of Entropy or hidden information about the integration complexity now vs before?
- Does the adopting of RESTful pattern to explicitly show context in APIs simplify this vs RPC style where the context is implicit in the data?
- What determines the complexity then?
- Number of nodes
- Number of edges
- Number of contexts per edge (nuanced or direct use)
- Amount of data per edge
- The type of edge – sync request/response, sync one-way, async one-way (event), async request/response
