Technology plays an increasingly important role in today’s society and facilitates how people interact with each other, how they learn and communicate. Artificial Intelligence (AI) solutions are constantly improving and are now outperforming humans in various tasks. AI algorithms have many areas of applications, including audio-visual perception, natural language processing (NLP), robotics and social intelligence.
However, a data scientist might not be able to efficiently optimise business processes from multiple domains alone. Often other engineers, experts in a particular area (e.g. designers, business analysts and process managers), are needed for an optimal solution.
One of the main concerns when implementing AI solutions in various businesses relates to the speed of delivering transitioning from a proof of concept prototype with relevant performance metrics towards a long-term operational contract with a client at scale.
The Amdaris adaptive data science accelerator
Amdaris created an AI accelerator from the observation that a data scientist might not be able to efficiently solve problems for businesses that require algorithms from a particular AI subfield that is not their speciality, such as computer vision, NLP, chemistry. The task will take a significantly longer amount of time and have an increased probability of being solved suboptimally.
Therefore, we required a framework that adapts tasks from a particular field of AI to another field of AI or business scenario.
The challenge of designing AI solutions
The main problems to tackle when designing AI solutions relate to model explainability, the potential for data bias and unawareness of risk. Additionally, on top of the core AI algorithms, the solution offered to a business partner probably needs to be packaged so that it can run on cloud computing resources to take advantage of their benefits of scale, cost and flexibility.
The ultimate end-user of the AI-enabled solution should be able to harvest the benefits of complex AI algorithms through endpoints that are agnostic of the software tools and hardware resources specific to the AI.
The Amdaris data science framework
Our AI framework has been developed according to the highest standards of delivering cloud-based software solutions.
Fig 1. AI application deployed on a cloud platform
As the main AI tools are available in Python, the Amdaris data science framework was developed in this programming language. But its design principles are valid for other programming languages too. The framework is built from multiple layers that are stacked on top of each other. Each layer contains multiple modules that have the same API interface to be used selectively to solve a problem.
Classical data science problems
For a classical data science problem, there are three categories of layers:
- Layers specific to the ETL (Extract Load Transform) of available data
- Layers specific to the business scenario
- Layers specific to AI solutions
The investigation starts with data transformation and understanding, usually with the help of a data engineer and business analyst. The data scientist or another AI specialist who then takes on the task of AI modelling will have a significant advantage of being able to focus on exploratory data analysis and also on providing enhanced AI model explainability.
The Amdaris AI accelerator was designed to embed the process of AI explainability in three stages:
- Pre-modelling explainability, through explainable feature engineering
- Explainable modelling, through the flexibility of employing different machine learning models for different parts of the same data instance
- Post-model explainability, by including modules using AI libraries for advanced explainability techniques, such as perturbation mechanisms
Amdaris’ customisable data science framework vs. AI cloud-based services
Amdaris AI accelerator
- Customised ML pipelines
- Compare multiple AI algorithms
- Consolidate business finances
- Keep sensitive data locally
- Seamless specialist collaboration
Cloud-based AI solutions
- Suboptimal performance
- Each algorithm adds extra cost
- Cost increases with the number of users
- Complicated deployment
- Difficult to switch cloud provider
Outcomes of the Amdaris data science framework
- Adapting domain-specific knowledge to run on generic data science toolchain, i.e. Pandas and Spark
- Highly parallelised design both for training and testing AI models in multithreaded mode
- Scalable by design on generic cluster technologies
- Integrating the latest version of AI libraries, e.g. TensorFlow 2.3, Scikit-learn 0.23 and Pandas 1.1
- The option of harmonising the accelerator with Kubernetes facilities, deployed on a cluster
Benefits of using the Amdaris data science framework
- The business scenario for a given project will benefit from a well-defined separation between the contributions of various AI specialists
- Two-way scaling capabilities are already built-in:
- Multithreading: applicable when dealing with small to medium size AI models
- Multi-serving: large size AI models can use AI serving
- There will be continuous integration of new business requirements due to the modular, layer-based plug and play architecture
- The application is platform-independent, running with identical behaviour on cloud or locally on servers with Windows or Linux operating systems
- It is standalone software (including visualisation), which results in enhanced transparency for business users and the application stakeholders through the vertical and horizontal governance process
What next?
If you would like to speak to someone at Amdaris about data science and AI, just get in touch.
Call +44(0)117 935 3444 or contact us using the form below and let us know about your next plans. We will help you choose the best technology for making your project a success.